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DEFINITIONS 

IDA  publishes  the  tallowing  documsnts  to  toport  the  results  of  Its  woili. 


Reports 

Reports  sro  the  most  suthoritatlM  and  most  caretally  considered  products  IDA  publishes. 
They  normally  embody  results  of  major  projects  which  |a)  hose  s  direct  bearing  on 
decisions  alfnctlng  major  programs,  (bj  address  Issues  of  significant  concern  to  the 
Executive  Branch,  the  Congress  and/or  the  public,  or  (c)  address  issues  that  have 
significant  economic  Implications.  IDA  Reports  are  reviewed  by  outside  panels  oi  axperts 
to  ensure  their  high  quality  and  reinvanca  to  the  problems  studied,  and  they  are  released 
by  the  President  of  IDA. 

Group  Reports 

Group  Reports  record  the  findings  and  rasutts  oi  IDA  sstablishad  working  groups  and 
panels  composed  ot  senior  individuals  addressing  major  issues  which  otherwise  would  ho 
the  subject  of  an  IDA  Report.  IDA  Group  Reports  are  leviswod  by  the  senior  individuals 
rasponsibla  tar  the  project  and  others  as  selected  by  IDA  to  ensure  their  high  quality  and 
reinvanca  to  the  problems  studied,  and  am  released  by  the  President  ol  IDA. 

Papers 

Papers,  also  authoritathrs  and  carefully  considered  products  ol  IDA,  address  studies  that 
are  narrower  in  scope  than  those  covered  in  Reports.  IDA  Papers  are  reviewed  to  ensure 
that  they  meet  the  high  standards  expected  of  retereed  papers  in  professional  journals  or 
formal  Agency  reports. 

Dpcuments 

IDA  Documents  are  used  tar  the  cnnveniance  Of  the  sponsors  or  the  analysts  (a)  to  record 
substantive  work  done  In  quick  reaction  studies,  (b)  to  record  the  proceedings  ol 
conlorences  and  mootings,  (c)  to  make  available  preliminary  and  tentative  results  of 
analyses,  (d)  to  record  data  daveloped  in  the  course  of  an  Investigation,  or  (e)  to  lorward 
intormatlon  that  is  essentially  unanalyzed  and  unevaluated.  The  review  of  IDA  Documents 
is  suited  to  their  content  and  intended  use. 
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INSTITUTE  FOR  DEFENSE  ANALYSES 

Contract  MDA  903  89  C  0003 
Task  T-R2-597.21 


PREFACE 


This  paper  was  prepared  by  the  Institute  for  Defense  Analyses  (IDA)  for  the 
Strategic  Defense  Initiative  Organization  (SDIO),  under  contract  MDA  903  89  C  0003, 
Subtask  Order  T-R2-597.21,  “SDS  Test  and  Evaluation.”  The  objective  of  the  subtask  is 
to  assist  the  SDIO  in  planning,  executing,  and  monitoring  software  testing  and  evaluation 
research,  development,  and  practice. 

In  support  of  this  objective,  IDA  conducted  an  examination  of  several  commer¬ 
cially  available  tools  that  support  static  and  dynamic  analysis  of  Ada  code.  This  paper 
presents  the  results  of  the  assessment  and  provides  software  development  managers  with 
information  on  current  capabilities  of  commercial  testing  tools. 

This  paper  was  reviewed  by  the  following  members  of  the  IDA  research  staff: 
Dr.  Robert  Atwell,  Dr.  Dennis  Fife,  Dr.  Randy  Garrett,  Dr.  Karen  Gordon,  Ms.  Audrey 
Hook,  and  Dr.  Richard  J.  Ivanetich. 


V 


EXECUTIVE  SUMMARY 


Software  testing  is  labor  intensive  and  can  consume  over  50%  of  software 
development  costs.  Rarely  is  sufficient,  effective  testing  performed  as  evidenced  by  the 
fact  that  a  failure  rate  of  3  to  10  failures  per  thousand  lines  of  code  is  typical  for  commer¬ 
cial  software.  Moreover,  the  cost  of  correcting  a  defect  increases  as  software  develop¬ 
ment  progresses;  for  example,  the  cost  of  fixing  a  requirements  fault  during  operation  can 
be  60  to  100  times  the  cost  of  fixing  that  same  fault  during  early  development  stages.  Con¬ 
sequently,  timely  defect  detection  is  important.  Automated  testing  tools  can  alleviate 
these  problems  by  reducing  the  traditionally  manual  nature  of  testing  and  encouraging  the 
application  of  improved  testing  practices. 

Over  one  hundred  testing  tools  from  nearly  seventy  vendors  of  testing  tools  were 
identified.  A  short  list  of  tools  supporting  the  static  and  dynamic  analysis  of  Ada  code 
was  developed.  Consideration  of  tools  that  are  limited  to  quality  analysis  or  regression 
analysis,  dependendent  on  special  hardware,  or  form  part  of  a  computer-aided  software 
engineering  system  was  postponed  and  these  tools  excluded  from  the  list.  From  the  short 
list,  ten  tools  that  support  the  testing  of  Ada  code  were  selected  for  examination:  MAL- 
PAS,  SoftTest,  S-TCAT/Ada,  TCAT/Ada,  TCAT-PATH,  TDGen,  TSCOPE,  TBGEN, 
TCMON,  and  TestGen.  During  the  course  of  the  examination,  the  tools  were  applied  to 
a  series  of  Ada  programs  in  order  to  assess  their  functionality.  Each  tool  was  then 
described  in  terms  of  its  functionality,  ease  of  use,  and  documentation  and  support.  Prob¬ 
lems  encountered  during  tool  use  and  other  pertinent  observations  were  also  recorded. 

Compared  to  other  software  development  tools  for  Ada,  the  market  offers  rela¬ 
tively  few  commercial  tools  that  support  the  static  and  dynamic  analysis  needed  for  test¬ 
ing  Ada  code.  Significant  findings  from  this  study  include  the  following: 

•  Some  of  the  examined  tools  could  be  brought  into  immediate  use  to  improve  the 
cost-effectiveness  of  testing  for  SDI  software  development  efforts. 

•  The  coverage  analyzers  provide  reporting  data  that  can  support  the  management 
of  SDI  testing  efforts. 

This  report  provides  software  development  managers  with  information  that  may 
help  them  gain  an  understanding  of  the  types  of  software  testing  tools  that  are  commer¬ 
cially  available  and  how  these  tools  can  aid  the  development  of  Ada  software. 
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1.  INTRODUCTION 

1.1  Purpose  and  Scope 

This  report  provides  software  development  managers  with  information  that  may 
help  them  gain  an  understanding  of  the  types  of  software  testing  tools  that  are  commer¬ 
cially  available,  the  functionality  of  these  tools,  and  how  they  can  aid  the  development  of 
Ada  software. 

Tools  are  available  to  support  a  variety  of  testing  tasks  at  different  stages  in  the 
software  life  cycle.  To  make  best  use  of  available  resources,  the  work  described  here  was 
limited  to  the  examination  of  tools  that  support  the  static  and  dynamic  analysis  needed 
for  testing  Ada  code.  Code-based  testing  was  selected  as  being  one  area  where  auto¬ 
mated  support  is  critically  needed  b  ^th  to  increase  software  reliability  and  to  reduce 
development  costs.  Restriction  to  the  Ada  programming  language  [ANSI/MIL- 
STD-1983]  was  adopted  in  view  of  Department  of  Defense  (DoD)  Directive  5000.1, 
which  requires  the  use  of  Ada  for  weapons  systems  [DoDD  1991].  The  Strategic  Defense 
System  Initiative  Organization  (SDIO)  [SDI  1991]  also  requires  use  of  Ada.  It  is 
expected,  therefore,  that  the  results  of  this  work  will  apply  to  the  majority  of  Strategic 
Defense  Initiative  (SDI)  software  development  efforts. 

Section  2  of  this  report  describes  how  particular  tools  were  selected  for  examina¬ 
tion  and  the  method  of  examination.  Details  of  the  capabilities  of  these  tools  and  obser¬ 
vations  made  during  the  examinations  are  given  in  Section  3.  The  findings  resulting  from 
this  work  are  reported  in  Section  4.  Appendix  A  lists  all  the  commercial  testing  tools  that 
were  identified.  Sample  outputs  from  each  examined  tool  are  provided  in  Appendices  B 
through  F. 

1.2  Background 

Definitions  of  some  relevant  terms  and  testing  statistics  will  help  to  clarify  the 
scope  of  this  work  and  the  applicability  of  the  findings  to  SDI  software  development 
efforts. 


An  error  is  a  mistake  made  by  a  software  developer.  Its  manifestation  may  be  a 
textual  problem  in  the  code  called  a  fault  or  defect.  A  failure  occurs  when  an  encoun¬ 
tered  fault  prevents  software  from  performing  a  required  function  within  specified  limits. 
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Testing  refers  to  the  act  of  detecting  the  presence  of  faults,  or  demonstrating  their 
absence,  and  is  distinguished  from  debugging  where  faults  are  textually  isolated.  Three 
stages  of  testing  are  commonly  recognized;  these  are  unit  testing,  integration  testing,  and 
system  testing.  In  the  first  of  these,  unit  testing,  each  program  module  is  tested  in  isola¬ 
tion.  In  integration  testing,  these  modules  are  combined  so  that  successively  larger 
groups  of  integrated  software  and  hardware  modules  can  be  tested.  Finally,  system  test¬ 
ing  examines  an  integrated  hardware  and  software  system  to  verify  that  the  system  meets 
its  specified  requirements. 

In  bottom-up  testing  the  modules  at  the  bottom  of  the  invocation  hierarchy  are 
tested  independently,  then  modules  at  the  next  higher  level  that  call  these  modules,  and 
so  on.  Top-down  testing  starts  at  the  highest-level  module,  with  stubs  replacing  the 
modules  it  invokes.  These  stubs  are  then  replaced  by  the  next  lower-level  modules,  with 
new  stubs  being  provided  for  modules  these  call,  and  so  on. 

This  report  looks  at  unit  and  integration  testing  by  both  dynamic  and  static  analy¬ 
sis.  Dynamic  analysis  approaches  rely  on  executing  a  piece  of  software  with  selected  test 
data.  The  effectiveness  of  any  dynamic  analysis  technique  is  directly  related  to  the  test 
data  used.  Current  tools  attempt  to  detect  faults  rather  than  demonstrate  the  absence  of 
faults.  Additionally,  these  tools  can  only  detect  faults  whose  effects  propagate  to  soft¬ 
ware  outputs.  Static  analysis  approaches  do  not  require  software  execution  and  can  dem¬ 
onstrate  the  absence  of  certain  types  of  defects  such  as  variable  typing  errors.  The 
absence  of  execution,  however,  means  that  they  cannot  detect  faults  that  depend  on  the 
underlying  operating  environment.  Consequently,  effective  testing  requires  a  combination 
of  static  and  dynamic  analysis  approaches.  The  key  types  of  dynamic  and  static  analysis 
tools  are  identified  in  Tables  1  and  2,  respectively.  Table  2  omits  tools  for  analyses  rou¬ 
tinely  performed  by  Ada  compilers,  such  as  type  analysis. 

For  dynamic  analysis,  different  strategies  or  heuristics  can  be  used  to  drive  test 
data  generation.  Commercial  automated  support  is  currently  only  available  for  func¬ 
tional  and  structural  strategies.  In  the  first  case,  test  data  is  derived  from  the  program’s 
requirements  with  no  regard  to  program  structure;  these  approaches  are  language-inde¬ 
pendent.  In  structural  strategies,  test  data  is  derived  solely  from  the  program  structure 
where  this  structure  is  often  represented  as  a  directed  graph  (digraph).  The  only  func¬ 
tional  strategy  currently  supported  is  cause-effect  graphing.  Here  causes  relate  to  distinct 
input  conditions  or  equivalence  classes  of  input  conditions,  and  effects  relate  to  the  result¬ 
ing  output  conditions  or  system  transformations.  Test  data  is  derived  from  a  combina¬ 
tional  logic  network  that  represents  the  logical  relationships  between  causes  and  effects. 
The  structural  strategies  at  the  unit  level  that  are  supported  are  branch  and  path  testing. 
Branch  testing  requires  each  conditional  branch  statement  and  the  code  segment  whose 
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TABLE  1.  Tools  to  Support  Dynamic  Testing 


Type 

Description 

Assertion  Analyzer 

During  execution,  evaluates  logical  expressions  inserted  into  the 
software  that  specify  required  program  states  or  conditions  on 
variables. 

Code  Instrumentor 

Inserts  probes,  such  as  instructions  or  assertions,  into  a  program  to 
aid  statement  or  resource  monitoring,  or  other  activities. 

Coverage  Analyzer 

Assesses  measures  associated  with  the  execution  of  program  structural 
elements  to  determine  the  adequacy  of  a  series  of  test  runs. 

Keystroke  Capture 

Captures  keystroke  sequences  for  automatic  playback  of  test  sessions. 

Mutation  Analyzer 

Determines  test  set  thorot^bness  by  measuring  the  extent  to  which  a 
test  set  can  discriminate  between  a  program  and  fault-simulating 
variants. 

Performance  Analyzer 

Measures  the  ability  of  a  (sub)system  to  perform  its  functions  within 
speed  and  storage  allocation  constraints. 

Oracle 

Produces  correct  outputs  to  compare  with  actual  software  outputs. 

Regression  Tester 

Retests  software  to  verify  that  modifications  have  not  caused 
unintended  effects  and  that  the  software  still  meets  specified 
requirements. 

Reliability  Analyzer 

Determines  achieved  level  of  reliability  of  an  existing  system 
(component)  based  on  the  rate  of  defect  detection. 

Test  Data  Generator 

Uses  a  test  strategy  to  generate  test  data  in  a  methodical  manner. 

The  ideal  is  to  find  the  minimal  set  of  test  cases  that  result  in 
discovery  of  a  maximal  set  of  defects. 

Test  Manager 

Controls  a  large  and  evolving  amount  of  information  on  system 
features  to  be  tested,  as  weU  as  test  plans,  test  data,  and  test 
results. 

Test  Bed 

Directs  the  execution  of  the  software  under  test  against  a  collection  of 
test  data  sets.  Usually  it  records  and  organizes  the  output  generated. 

Trace  Analyzer 

Provides  a  record  of  program  execution;  it  states  the  sequence  in 
which  instructions  were  executed. 
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TABLE  2.  Tools  to  Support  Static  Analysis 


Type  Description 

Code  Auditor  Checks  for  conformance  to  prescribed  programming  standards  and 

practices. 

Concurrency  Analyzer  Determines  synchronization  patterns  in  a  concurrent  program. 

Cross-Reference  Analyzer  Provides  cross-reference  information  on  system  components,  e.g. , 

data  name,  statement  label,  literal  use,  inter-subroutine  call 
cross-indexes. 

Data  Flow  Analyzer  Performs  graphical  analysis  of  collections  of  sequential  data  definitions 

and  reference  patterns  to  determine  constraints  that  can  be  placed  on 
data  values  at  various  execution  points. 

Expression  Analyzer  Detects  certain  commonly  occurring  faults  associated  with  evaluation 

of  expressions. 

Interface  Analyzer  Checks  the  interfaces  between  program  units  for  consistency  and 

adherence  to  predefined  rules  or  axioms. 

Metric  Analyzer  Measures  the  extent  or  degree  to  which  a  product  possesses  and 

exhibits  a  certain  quality,  property,  or  attribute. 

Path  Analyzer  Identifies  all  possible  paths  through  a  program  to  detect  incomplete 

paths,  unexecutable  paths,  or  conditions  that  drive  path  execution. 

Reference  Analyzer  Detects  reference  anomalies,  e.g.,  when  a  variable  is  referenced 

along  a  program  path  before  it  is  assigned  a  value. 

Safety  Analyzer  Identifies  the  possible  causes,  and  consequence,  of  critical  system 

failures  and  so  determines  the  necessary  fault  tolerance  or  other 
mechanisms  needed  to  ensure  safe  operation  under  various 
operating  conditions. 

Symbolic  Evaluator  Accepts  symbolic  values  for  some  program  inputs  and  algebraically 

manipulates  them  according  to  the  expressions  in  which  they  appear. 

Unit  Analyzer  Determines  whether  or  not  the  units  or  physical  dimensions  attributed 

to  an  entity  are  correctly  defined  and  consistently  used. 


Update  Analyzer 


Compares  two  versions  of  a  module  to  look  for  differences. 


execution  is  controlled  by  this  conditional  to  be  executed  at  least  once.  Similarly,  path 
testing  requires  execution  of  every  path  conditional  and  associated  code  segment.  Path 
testing  is  the  more  stringent  strategy  but  can  incur  unacceptable  computational  costs.  The 
equivalent  structural  strategy  at  the  integration  level  requires  that  each  pair  of  module 
invocations  is  executed  at  least  once. 

Testing  is  labor-intensive  and  can  consume  over  50%  of  software  development 
costs.  In  one  particular  case,  NASA’s  Apollo  program,  80%  of  the  total  software  devel¬ 
opment  effort  was  incurred  by  testing  [Duim  1984].  In  general,  schedule  pressure  limits 
the  amount  of  testing  that  is  performed  and  defects  frequently  lead  to  the  failure  of  opera¬ 
tional  software.  For  example,  3  to  10  failures  per  thousand  lines  of  code  (KLOC)  are 
typical  for  commercial  software  and  1  to  3  failures  for  industrial  software  [Boehm  1988]. 
With  a  rate  of  0.1  failures  per  KLOC  after  delivery  for  its  shuttle  code,  however,  NASA 
has  demonstrated  that  poor  reliability  can  be  avoided  [Myers  1988].  The  cost  of  correct¬ 
ing  a  defect  increases  as  software  development  progresses;  for  example,  the  cost  of  fixing 
a  requirements  fault  during  operation  can  be  60  to  100  times  the  cost  of  fixing  that  same 
fault  during  early  development  stages  [Pressman  1987].  Consequently,  timely  defect 
detection  is  important. 

Automated  tools  can  improve  testing  cost  effectiveness;  indeed,  they  are  a  pre¬ 
requisite  for  most  static  analysis  approaches.  In  addition  to  eliminating  some  repetitive 
manual  tasks,  tools  can  promote  effective  dynamic  analysis  by  guiding  the  selection  of  test 
data  and  monitoring  test  executions.  Through  capturing  and  reporting  data  gathered  dur¬ 
ing  the  performance  of  testing  activities,  tools  also  support  the  quantitative  process  mea¬ 
surement  that  is  necessary  for  controlling  the  testing  process.  Benefits  claimed  by  some 
of  the  tools  discussed  later  include;^ 

•  Tool  X  can  save  a  developer  $15,000  or  more  per  KLOC. 

•  Tool  Y  has  given  clients  an  8:1  reduction  in  test  development  effort  and  one  client 

has  achieved  a  reduction  from  1.3  to  0.072  failures  per  1,000  lines  of  source  code. 

Of  course,  testing  tools  are  not  the  only  mechanism  for  improving  software  relia¬ 
bility.  Software  inspections,  for  example,  have  been  reported  to  find  60%  to  90%  of  soft¬ 
ware  defects,  while  reducing  total  development  costs  by  as  much  as  25%  [Fagan  1986]. 
These  and  other  approaches  are  discussed  in  [Brykczynski  1990]. 


1.  Tool  names  are  omitted  since  these  claims  have  been  neither  validated,  nor  invalidated,  in  the  course  of 
this  work. 
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2.  APPROACH  AND  METHODS 

The  overall  approach  taken  to  this  work  was  to  identify  vendors  of  testing  tools, 
select  tools  for  examination,  and  apply  the  selected  tools  in  the  testing  of  sample  pieces 
of  code.  These  activities  are  described  below. 

2.1  Vendor  Identification 

Tool  vendors  were  identified  from  a  number  of  sources,  specifically: 

•  The  tools  fair  reported  in  IEEE  Software  [Lutz  1990]. 

•  The  survey  of  Ada  tool  and  service  suppliers  in  Defense  Science  [DefSci  1990]. 

•  EDA’s  survey  of  the  state  of  the  art  in  software  testing  and  analysis  [Youngblut 
1989]. 

•  Input  from  the  SDI  Cooperative  Research  Exchange  (SCORE)  effort. 

•  Tools  exhibited  at  the  Tri-Ada  Conference,  held  October  1990  in  Baltimore. 

•  The  tools  fair  at  the  8th  International  Conference  on  Testing  Computer  Software, 
held  July  1991  in  Washington  D.C. 

Nearly  seventy  vendors  of  over  a  hundred  tools  were  identified.  A  list  of  these  is  given  in 
Table  A-1  in  Appendix  A.  A  short  list  of  those  tools  supporting  static  and  dynamic  anal¬ 
ysis  of  Ada  code  was  prepared  and  tool  information  sought  from  the  vendors.  In  several 
cases,  vendors  gave  in-house  demonstrations  of  their  tools. 

2.2  Tool  Selection 

Additional  criteria  were  applied  to  refine  the  list  to  be  compatible  with  the 
resources  available  for  tool  examination.  To  ensure  that  the  results  apply  to  the  largest 
possible  audience,  it  was  determined  that  selected  tools  should  be  essentially  independent 
of  processor  architecture.  Consequently,  non-intrusive  monitors  for  real-time  systems, 
which  require  special  purpose  hardware,  were  not  considered.  For  a  similar  reason,  tools 
tied  to  a  particular  Ada  compiler  were  not  considered.  Because  the  relationship  between 
quality  metrics  and  software  reliability  is  not  well  understood,  tools  restricted  to  quality 
analysis  were  also  not  considered.  Examination  of  testing  tools  that  are  part  of  a  com¬ 
puter-aided  software  engineering  (CASE)  system  or  that  perform  regression  testing  was 


7 


postponed  for  a  later  effort. 

The  following  tools  were  selected: 

•  Malvern  Program  Analysis  System  (MALPAS).  MALPAS  provides  static  analy¬ 
sis  capabilities  suitable  for  unit  testing  and  bottom-up  integration  testing.  It  uti¬ 
lizes  an  intermediate  language  (IL).  Translators  exist  for  several  languages 
including  Pascal,  Fortran,  and  C. 

•  SoftTest.  This  is  a  requirements-based  testing  tool  that  uses  cause-effect  graphing 
to  generate  test  conditions.  It  is  independent  of  programming  languages. 

•  TCAT/Ada,  TCAT-PATH,  S-TCAT/Ada,  TDGen,  and  TSCOPE.  These  tools 
provide  structural  coverage  analysis  at  the  unit  and  integration  levels,  test  data 
generation,  and  animation  of  the  increasing  levels  of  coverage  achieved.  Ver¬ 
sions  of  the  tools  supporting  Ada,  C,  Cobol,  Fortran,  and  Pascal  programming 
languages  are  available. 

•  Test  Bed  Generator  (TBGEN)  and  Test  Coverage  Monitor/Program  Bottleneck 
Finder  (TCMON).  TBGEN  supports  dynamic  testing  of  Ada  code  at  the  unit 
and  integration  levels  by  generating  a  testbed  that  allows  a  user  to  control  subpro¬ 
gram  execution  and  observe  the  results.  TCMON  allows  a  user  to  study  the  cov¬ 
erage  of  test  data  or  analyze  the  dynamic  behavior  of  an  Ada  program.  These 
two  tools  can  be  used  independently  or  combined  to  generate  a  monitored 
testbed. 

•  TestGen.  Part  of  the  Ada  Integrated  Software  Lifecycle  Environment  (AISLE) 
family  of  Ada  design  tools,  TestGen  provides  three  types  of  structural  module 
coverage  analysis  for  testing  Ada  code  at  the  unit  level. 

Two  additional  selected  tools  whose  examinations  were  postponed  are: 

•  AdaQuest.  A  static  and  dynamic  testing  system  based  on  an  existing  verification 
and  validation  system  for  Fortran.  The  first  complete  version  of  AdaQuest  was 
due  for  release  in  May  1991  but  is  unavailable  as  of  this  writing. 

•  T.  A  new  version  of  this  functional  test  data  generation  tool  that  includes  exten¬ 
sive  additions  is  due  to  be  released  in  September  1991.  A  testbed  that  drives  pro¬ 
gram  execution  is  expected  to  be  released  at  the  same  time.  Examination  of  T 
has  been  postponed  until  these  become  available. 


2.3  Method  of  Examination 

Each  tool  was  used  to  test  several  small  Ada  programs.  The  goal  of  these  initial 
tool  applications  was  to  allow  the  examiner  to  gain  familiarity  with  overall  tool  operation. 
Each  tool  was  subsequently  applied  to  the  same  Ada  program.  This  software  was  the 
Ada  Lexical  Analyzer  Generator  program  that  creates  a  lexical  analyzer  or  “next-token” 
procedure  for  use  in  a  compiler,  pretty  printer,  or  other  language  processing  program 
[Meeson  1989].  It  was  developed  for  the  Software  Technology  for  Adaptable,  Reliable 
Systems  (STARS)  program  and  consists  of  several  Ada  subprograms  with  a  total  of  3,253 
lines  of  code. 

The  experience  gained  by  installing  and  using  the  tools  was  used  to  prepare  a 
short  review  of  each  tool.  Determination  of  the  appropriate  high-level  information  to  col¬ 
lect  was  based  on  questions  given  in  the  Software  Engineering  Institute’s  A  Guide  to  the 
Classification  and  Assessment  of  Software  Engineering  Tools  [Firth  1987],  and  Brown- 
stein  and  Lerner’s  Guidelines  for  Evaluating  and  Selecting  Software  Packages  [Brown- 
stein  1982].  More  detailed  information  requirements  were  deduced  from  the  characteris¬ 
tics  of  the  tools  themselves. 
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3.  TOOL  EXAMINATIONS 


This  section  describes  the  selected  tools  in  terms  of  pertinent  vendor  details, 
operating  environments,  and  the  functionality  provided.  Price  information  accurate  at 
the  time  of  examination  is  also  included.  Each  description  is  supported  with  observations 
that  discuss  ease  of  use,  documentation  and  user  support,  and  Ada  restrictions.  Brief 
mention  of  any  problems  encountered  during  the  examinations  provides  insight  into  the 
reliability  and  robustness  of  each  tool. 

3.1  MALPAS 

MALPAS  comprises  a  suite  of  static  analyzers  that  provide  control  flow,  data 
use,  input/output  dependency,  and  complexity  analysis  and  symbolic  execution. 

3.1.1  Tool  Overview 

MALPAS  was  developed  in  the  late  1970s  at  the  United  Kingdom  Ministry  of 
Defense  Royal  Signals  and  Radar  Establishment  to  verify  avionics  and  other  safety-criti¬ 
cal  defense  system  software.  Since  1986  it  has  been  marketed  and  supported  by  Rex, 
Thompson  &  Partners  (RTP).  MALPAS  has  50  users,  including  5  Ada  sites.  The  Ada 
translator  is  a  new  product  released  in  July  1991.  RTP  also  markets  seminars  to  intro¬ 
duce  potential  customers  to  MALPAS  and  training  courses.  A  users  group  is  supported. 
MALPAS  is  available  on  VAXA^S  platforms.  The  tools  examined  in  this  study  were 
MALPAS  Release  5.1,  IL  Version  5,  Pascal-IL  Translator  3.1,  and  Ada-IL  Translator 
1.01.  The  price  for  MALPAS  and  the  Ada-IL  translator  at  the  time  was  $60,000. 

The  analyses  performed  by  MALPAS  are  intended  to  assure  software  safety,  reli¬ 
ability,  consistency,  and  conformance  to  standards.  They  include  the  following: 

•  Control  flow  analysis  to  reveal  underlying  program  structure,  unreachable  code. 

•  Data  flow  analysis  to  detect  uninitialized  variables,  successive  assignments  with¬ 
out  use. 

•  Information  flow  analysis  to  identify  input-output  dependencies. 

•  Path  assessment  to  produce  a  structural  complexity  measure. 
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•  Partial  analysis  using  program  slicing  to  reduce  analysis  time. 

•  Semantic  analysis  to  provide  symbolic  execution  for  each  loop-free  path. 

•  Compliance  analysis  to  verify  code  against  formal  specifications. 

MALPAS  analyses  are  based  on  an  Intermediate  Language  (IL)  representation 
of  program  specifications  or  source  code.  Translators  from  several  languages  (including 
Ada,  C,  Fortran,  and  Pascal)  to  IL  are  available.  The  approach  of  using  a  common 
intermediate  language  for  analyses  simplifies  the  extension  of  MALPAS’s  capabilities  to 
other  programming  languages.  Formal  program  specifications  can  also  be  expressed  in 
IL.  At  present,  however,  no  automated  translation  tools  for  other  formal  specification 
languages  such  as  OBJ,  Vienna  Development  Method  (VDM),  or  Z  are  supported. 

Analyzing  application  source  code  is  a  two-step  process.  First  the  code  is 
translated  into  IL.  Since  the  Ada  translator  was  not  available  when  the  tool  examination 
started,  the  Pascal  translator  was  examined  first.  Pascal  code  is  translated  as  a  single 
complete  program;  this  is  a  straightforward  process.  The  translation  of  Ada  source  code 
to  IL  is  significantly  more  complicated.  The  sample  Ada  code  analyzed  contained  sev¬ 
eral  separately  compiled  packages  and  subunits.  First  the  generic  input/output  packages 
used  by  the  program  had  to  be  instantiated  (by  hand),  translated,  and  loaded  into  an  IL 
code  library.  Then  each  program  unit  had  to  be  translated  and  loaded  into  the  IL  code 
library. 

The  second  step  is  to  run  the  analyses  on  the  IL  code.  A  smgle  tool  controls  all  of 
the  available  analyses.  Options  are  selected  by  command  line  parameters  and  results  are 
written  to  files  that  can  be  printed.  Default  parameter  settings  for  initial  analyses  of  new 
code  were  set  up  to  include  control  flow,  data  use,  and  information  flow  analyses.  The 
compliance  and  semantic  analyses  are  computationally  more  complex.  The  partial  anal¬ 
ysis  capability  allows  these  analyses  to  be  restricted  to  particular  modules  or  paths  within 
the  program. 

Control  flow,  data  flow,  and  information  flow  analyses  are  fairly  standard  static 
analysis  techniques.  Structured  programming  has  largely  eliminated  control  flow 
anomalies.  Data  flow  and  information  flow  anomalies,  however,  are  still  useful  indica¬ 
tors  of  potential  problems.  Information  flow,  for  example,  identifies  all  of  a  subprogram’s 
inputs  and  outputs,  which  may  be  more  than  those  passed  as  parameters.  MALPAS’s 
semantic  analysis  option  provides  symbolic  execution  of  loop-free  code  segments.  That 
is,  for  each  possible  path  through  a  segment,  the  value  of  each  modified  variable  is  given 
as  an  algebraic  expression  in  terms  of  the  input  variables.  This  provides  valuable  feed¬ 
back  to  a  programmer  about  the  meaning  of  the  code  and  the  results  that  will  be  produced 
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when  the  code  is  executed.  The  compliance  testing  option  uses  this  same  information  to 
check  formally  specified  requirements  that  have  been  added  to  the  IL  code. 

3.1.2  Observations 

Ease  of  use.  MALPAS  is  a  batch-oriented  tool  even  though  it  may  be  invoked 
interactively.  The  only  user  interaction  is  through  the  set  of  options  that  can  be  selected 
from  the  command  line.  The  large  number  of  options  may  make  MALPAS  “flexible”  for 
expert  users.  Novice  or  casual  users,  however,  may  have  some  difficulty  controlling  non¬ 
default  processing. 

Introducing  the  intermediate  language  for  analyses  may  cause  problems  for  some 
users.  All  analyses  and  reports  refer  to  the  IL  version  of  the  program  rather  than  to  the 
original  source  code.  The  mapping  back  to  the  original  code  must  be  done  manually.  The 
intermediate  language  approach  may  simplify  extending  MALPAS  to  cover  a  range  of 
different  programming  languages  (by  requiring  only  new  IL  translators),  but  it  imposes  a 
level  of  separation  between  the  actual  source  code  and  the  analyses  that  must  be  compen¬ 
sated  for  by  the  user. 

Translating  Ada  source  code  to  IL  was  found  to  be  somewhat  more  complicated 
than  expected.  The  sample  Ada  code  analyzed  contained  several  separate  packages  and 
subunits,  and  normally  requires  several  compilation  steps.  The  MALPAS  Ada  to  IL 
translator,  however,  required  several  additional  steps  that  Ada  compilers  either  do  not 
need  or  are  able  to  hide. 

Documentation  and  user  support.  Installation  and  operating  instructions  were 
clear,  thorough,  and  accurate.  Installation  required  simply  editing  sample  conunand  files 
to  name  local  directories  and  disks.  The  manuals  included  good  examples  and  the  tools 
operated  exactly  as  described. 

Ada  restrictions.  Although  support  for  all  aspects  of  Ada  that  can  be  analyzed 
statically  is  the  vendor’s  eventual  goal,  the  current  MALPAS  tools  support  only  a  subset 
of  Ada.  The  Ada  to  IL  translator  recognizes  all  valid  Ada  code  but  the  translation  to  IL 
is  not  complete.  The  intermediate  language,  for  example,  does  not  include  any  mecha¬ 
nism  for  concurrency,  so  Ada  tasks  cannot  be  translated.  This  restriction  is  particularly 
unfortunate  because  execution-based  testing  of  concurrent  programs  is  often  difficult  to 
control.  Repeating  a  particular  test,  for  example,  might  not  produce  the  same  results 
each  time.  Rigorous  static  analyses  of  potential  task  interactions  would  contribute  signifi¬ 
cantly  to  identifying  and  correcting  tasking  problems. 

Translation  of  Ada’s  generic  program  units  is  not  supported.  Generic  units  pro¬ 
vide  a  powerful  mechanism  that  simplifies  programs  and  enhances  reuse.  Ada’s  standard 
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input  and  output  facilities,  for  example,  are  defined  in  terms  of  generic  packages.  MAL- 
PAS  currently  requires  manual  instantiation  of  any  required  generic  units. 

Access  types  (pointers)  and  dynamic  storage  allocation  are  not  supported.  Anal¬ 
ysis  of  unconstrained  use  of  pointers,  for  example  to  detect  potential  “dangling”  pointers, 
is  virtually  impossible.  A  workaround  for  disciplined  use  of  pointers  for  data  structures 
such  as  linked  lists  is  to  define  abstract  data  types  that  encapsulate  the  pointers.  MAL- 
PAS  would  be  able  to  analyze  application  code  that  used  the  abstract  data  types  since  the 
pointers  are  hidden.  MALPAS,  however,  would  not  be  able  to  analyze  an  implementa¬ 
tion  of  the  abstraction  that  used  pointers. 

Problems  encountered.  The  MALPAS  tools  performed  as  specified  in  their  docu¬ 
mentation.  No  failures  occurred  in  use. 

3.2  SoftTest 

SoftTest  supports  requirements-based  test  analysis  using  cause-effect  graphing.  It 
derives  test  conditions  to  guide  the  preparation  of  test  data.  It  also  provides  a  measure  of 
test  adequacy  in  terms  of  the  number  of  testable  functional  variations  for  which  tests  have 
been  specified. 

3.2.1  Tool  Overview 

SoftTest  was  developed  in  1987  and  is  marketed  and  supported  by  Bender  and 
Associates.  There  are  currently  over  50  users.  The  tool  executes  on  any  IBM  PC,  XT, 
AT,  PS2  or  compatible  platform  under  MS-DOS.  Bender  also  markets  project  methodol¬ 
ogy  guidelines,  consulting  services,  and  training  courses  on  software  quality  assurance 
and  testing.  The  version  of  SoftTest  examined  was  Release  3.1.  At  the  time  the  price 
was  $2,500. 

SoftTest  automates  a  requirements  analysis  technique  called  cause-effect  graph¬ 
ing,  developed  at  IBM  in  the  early  1970s.  The  primary  phases  of  analysis  are  as  follows: 

•  Extraction  of  node,  relation,  and  constraint  definitions  from  cause-effect  graphs. 

•  Functional  variation  analysis  to  identify  combinations  of  input  conditions 

required  for  tests. 

•  Test  condition  synthesis  to  consolidate  variations  and  produce  minimal  test  sets. 

A  cause-effect  graph  identifies  the  set  of  input  conditions  (the  causes)  that  a  pro¬ 
gram  must  respond  to,  and  relates  these  to  the  set  of  output  conditions  (the  effects)  that 
the  program  must  produce.  Relations  between  inputs,  outputs,  and  intermediate  condi¬ 
tions  are  specified  in  terms  of  combinational  logic  (AND,  OR,  NOT).  Special 
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relationships  such  as  mutually  exclusive  input  conditions  and  conditions  that  hide  or  mask 
other  conditions  can  also  be  specified. 

The  functional  variations  of  a  cause-effect  graph  reflect  all  the  individual  unique 
functions  the  software  is  required  to  perform.  The  thesis  of  this  approach  to  testing  is 
that  although  the  number  of  possible  combinations  of  input  conditions  may  be  very  large, 
a  program  can  be  thoroughly  tested  by  exercising  this  relatively  small  set  of  unique  func¬ 
tions. 


Some  functional  variations  may  not  be  testable  because,  for  example,  it  may  be 
physically  impossible  for  certain  combinations  of  input  conditions  to  arise.  Another  rea¬ 
son  is  because  the  results  of  one  function  may  be  obscured  by  other  functions.  When  this 
latter  case  arises,  SoftTest  identifies  intermediate  program  results  that,  if  they  could  be 
observed,  would  enable  otherwise  obscured  functions  to  be  tested. 

A  single  test  may  exercise  several  functions.  This  means  that  a  smaller  number 
of  test  cases  will  often  be  able  to  exercise  all  of  a  program’s  functionality.  SoftTest 
includes  analysis  that  yields  a  minimal  number  of  test  cases  that  will  exercise  all  the 
testable  functional  variations.  Additional  tests  for  special  cases  such  as  boundary  condi¬ 
tions  can  be  added  to  the  tests  produced  by  SoftTest. 

Cause-effect  graphs  express  only  combinational  relationships,  constructed  from 
AND,  OR,  and  NOT  operations,  between  causes  and  effects.  Graphs  are  not  allowed  to 
form  loops  that  connect  effects  back  to  causes.  This  raises  a  question  about  how  to  test 
programs  that  clearly  require  iterative  or  recursive  processing.  For  example,  the  func¬ 
tionality  of  sorting  a  fixed  number  of  inputs  can  be  specified  using  AND,  OR,  and  NOT; 
but  sorting  arbitrary  length  lists  or  files  of  inputs  cannot.  By  unrolling  the  first  few  itera¬ 
tions  of  the  (assumed)  loop  structure,  SoftTest  can  generate  test  conditions  that  achieve 
virtually  any  level  of  coverage.  That  is,  if  the  functionality  required  could  be  imple¬ 
mented  by: 

while  not  completed  loop 
ioop_body; 
end  loop; 

then  the  cause-effect  graph  could  be  specified  as  if  the  implementation  would  be; 

if  not  completed  then 

loop^ody;  —  first  iteration 
if  not  completed  then 

loop_body;  —  second  iteration 
••• 

end  if; 
end  if; 
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The  test  conditions  produced  for  this  specification  cover  zero,  one,  and  two  iterations  of 
the  loop  body,  which  would  exercise  every  combination  of  pairwise  linear  code  segments. 
Although  not  exhaustive,  this  level  of  coverage  is  usually  considered  fairly  thorough  test¬ 
ing.  Additional,  “realistic”  tests  for  sorting,  for  example,  could  be  added  to  the  tests  pro¬ 
duced  by  this  analysis.  Requirements-based  testing,  of  course,  is  not  supposed  to  use 
knowledge  of  a  program’s  structure.  The  required  functionality,  however,  can  strongly 
indicate  an  iterative  (or  recursive)  implementation. 

3.2.2  Observations 

Ease  of  use.  SoftTest’s  user  interface  provides  simple  menu-driven  commands  to 
initiate  processing  and  review  results.  It  is  also  possible  to  invoke  one  of  a  number  of 
third-party  text  editors  from  within  the  tool  so  that  graph  specifications  can  be  corrected 
and  analyses  rerun  without  leaving  the  tool.  Analysis  reports  can  also  be  easily  printed. 
The  hard  part  about  using  SoftTest  is  developing  complete  functional  specifications  for 
software  to  be  tested.  Even  though  the  cause-effect  graph  language  is  clear  and  simple, 
writing  specifications  in  this  form  requires  some  experience.  New  users  should  expect  to 
spend  some  time  with  the  tutorial  materials  provided.  Training  courses  offered  by  the 
vendor  may  also  prove  useful. 

Documentation  and  user  support.  Ti»e  tool  documentation  and  user  support  were 
quite  good.  Installation  was  simple  and  the  tool  operated  exactly  as  described  in  the  ref¬ 
erence  manual.  Two  tutorials  were  provided-one  that  worked  through  examples  of  how 
to  run  the  tool  and  one  that  discussed  requirements-based  testing  in  more  general  terms. 
Bender  and  Associates  answered  several  questions  about  cause-effect  graphing  tech¬ 
niques  over  the  phone. 

Ada  restrictions.  SoftTest  is  independent  of  a  particular  programming  language. 

Problems  encountered.  SoftTest  performed  as  documented.  No  problems  were 
encountered  in  its  use. 

3.3  TCAT/Ada,  TCAT-PATH,  TDGen,  S-TCAT/Ada,  and  TSCOPE 

These  tools  are  part  of  the  Software  TestWorks  toolset  that  also  includes 
SMARTS,  CAPBAK,  and  EXDIFF  for  regression  testing.  Two  more  tools,  SpecTest 
and  MetaTest,  are  planned  for  release  late  1991  and  will  use  software  specifications  and 
designs  to  guide  code  testing. 

3.3.1  Tool  Overview 

Software  TestWorks  has  been  marketed  by  Software  Research  for  over  five  years. 
Each  type  of  coverage  analyzer,  that  is  TCAT,  TCAT-PATH,  has  respectively  2,100, 
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1,800,  and  1,600  users.  TDGen  has  over  400  users,  and  TSCOPE  has  1,100  users.  Soft¬ 
ware  Research  also  offers  a  range  of  software  testing  services,  technical  seminars,  and 
programming  language  validation  suites.  The  tools  are  available  on  operating  platforms 
ranging  from  personal  computers  (PCs)  to  mainframes  under  UNIX,  MS-DOS,  OS-2, 
and  VMS  operating  systems.  TSCOPE  requires  X-Windows.  Prices  dependent  on  the 
operating  environment  and  start  at  $1,900  for  TCAT/Ada,  $1,950  for  TCAT-PATH,  $800 
for  S-TCAT/Ada,  $30^*  for  TDGen,  and  $900  for  TSCOPE.  The  examinations  were  per- 
formcd  on  a  Sun-4  copy  of  the  TCAT/Ada  Version  7.6,  S-TCAT/Ada  Version  7.6, 
TCAT-PATH  Release  7,  TDGen  Release  3.2,  and  TSCOPE  Release  1.2. 

TCAT/Ada  instruments  the  code  contained  in  a  user-specified  list  of  files  to 
reveal  whether  each  module  branch  has  been  executed.  This  process  also  produces  a 
program  listing,  called  a  reference  listing,  that  shows  the  location  and  numbering  of 
markers  that  identify  particular  code  segments.  The  user  then  compiles  the  instrumented 
program  and  links  it  with  a  provided  runtime  file.  When  run,  this  program  queries  the 
user  for  the  name  of  the  tracefile  to  which  execution  data  will  be  written.  This  tracefile  is 
subsequently  used  by  the  reporting  utility  to  list  the  overall  coverage  achieved,  identify  hit 
and  not-hit  segments,  and  produce  histograms  showing  the  frequency  distribution  of  seg¬ 
ments  hit  using  either  linear  or  logarithmic  scales.  All  segment  information  is  given  in 
terms  of  the  segment  numbers  shown  in  the  reference  listing.  The  reporting  utility  can 
combine  archived  test  data  with  new  tracefile  data  if  desired.  At  the  end  of  each  run,  the 
tracefile  is  combined  with  the  specified  archive  file  to  produce  a  new  archive  file.  With 
the  exception  of  information  on  the  sequence  in  which  segments  were  hit,  archive  files 
contain  the  same  data  as  a  tracefile.  For  integration  testing,  a  special  utility  that  moni¬ 
tors  the  total  could-have-been-hit  count  is  provided  to  prevent  initial  tests,  that  may  not 
touch  all  program  modules,  from  producing  artificially  high  coverage  counts.  Additional 
utility  programs  for  abbreviated  coverage  reports,  reporting  coverage  on  individual 
modules,  and  analyzing  trace  files  for  record  types  are  provided. 

S-TCAT/Ada  provides  the  same  basic  functionality  for  analyzing 
module-to-module  interfaces  that  have  been  exercised  as  TCAT/Ada  provides  for 
branches.  Modules  are  defined  to  be  all  Ada  procedures  and  functions,  except  runtime 
functions  and  those  appearing  in  a  user-specified  list  of  interfaces.  An  additional  utility  is 
provided  to  generate  a  tabular  representation  of  the  call  graph  of  the  source  program. 

TCAT-PATH  is  also  similar  to  TCAT/Ada  except  that  coverage  reporting 
addresses  the  paths  executed  and  is  only  provided  on  a  single  tracefile;  archive  files  are 
not  supported.  In  addition  to  the  instrumented  code  and  reference  listing,  the  instrumen- 
tor  generates  a  separate  digraph  for  each  module.  This  can  be  used  to  count  the  number 
of  paths  and  display  path  statistics.  A  set  of  module  paths  also  can  be  generated  from  a 
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digraph  with  the  user  li-niting  path  generation  or  preventing  the  output  of  some  generated 
paths  as  necessary.  Additional  utilities  are  available  to  generate  an  (approximate)  pic¬ 
ture,  determine  the  cyclomatic  complexity  of  a  digraph,  and  support  use  of  TCAT-PATH 
th  UNIX-type  make  files. 

TSCOPE  can  be  used  with  TCAT/Ada,  TCAT-PATH,  or  S-TCAT/Ada  to  ani¬ 
mate  test  completeness.  Each  program  module  can  be  instrumented  for  either  branch, 
path  coverage,  or  call-pair  coverage.  All  instrumented  modules  are  reported  on  a  single 
display  and  different  kinds  of  reporting  can  be  selected  for  different  modules.  Graphic 
representations  such  as  histograms  are  available  for  dynamic  displays.  Digraphs  and 
call-trees  can  be  displayed  either  dynamically  or  statically.  TSCOPE  employs  a  graphical 
user  interface  and  output  displays  are  mapped  to  an  X-Window  screen  by  means  of  a 
user-specified  configuration  file. 

TDGen  works  with  two  files.  The  template  file  tells  TDGen  how  to  generate  test 
data  based  on  data  supplied  in  a  values  file.  TDGen  replaces  variable  fields,  called 
descriptors,  in  the  template  with  values  from  the  values  file.  Descriptors  may  be  user- 
defined  or  take  one  of  the  predefined  values  (ascii,  alpha,  decimal,  and  real).  When 
invoking  TDGen  the  user  specifies  whether  values  for  the  descriptors  should  be  taken 
from  integers  given  in  the  command  line,  randomly  from  the  values  file,  or  sequentially 
from  the  values  file  to  generate  every  possible  combination  of  values.  To  aid  this  choice, 
TDGen  can  be  requested  to  tabulate  the  number  of  possible  test  data  combinations.  A 
provision  for  escaping  to  the  operating  system  level  is  provided  to  allow,  for  example, 
editing  files  during  a  TDGen  session. 

3.3.2  Observations 

Ease  of  use.  A  user  can  interact  with  these  tools  using  either  a  conunand-line 
interface  or  a  series  of  menus.  Only  the  command-line  interface  versions  of  the  tools 
were  available  for  the  examination  reported  here.  In  the  case  of  TCAT-PATH,  support 
for  UNIX-like  make  files  is  provided  to  facilitate  repetitive  compilation  and  linking  tasks. 
Context-sensitive  help  and  help  frames  discussing  each  function  are  provided.  No  special 
knowledge  is  required  to  use  these  tools. 

Reports  are  generally  well-structured.  Since  segments,  paths,  and  call-pairs  are 
referred  to  by  number,  however,  a  user  must  refer  back  to  the  various  reference  listings  to 
identify  the  subject  of  each  reference.  Path  descriptions,  given  in  terms  of  numbered 
edges  and  nodes  from  the  underlying  digraph,  are  particularly  difficult  to  read.  The  pre¬ 
sentation  of  histograms,  digraphs,  and  call  graphs  would  benefit  from  the  use  of  modern 
graphical  techniques. 


Documentation  and  user  support.  The  tools  are  supported  by  extensive  documen¬ 
tation  that  includes  guidelines  on  appropriate  minimum  coverage  levels.  Examination  of 
these  tools  was  delayed  by  the  time  it  took  the  vendor  to  respond  to  tool  problems. 

Instrumentation  Overhead.  TCAT/Ada  and  TCAT-PATH  instrument  the  con¬ 
tents  of  files  specified  as  part  of  the  tool  invocation.  In  each  case,  all  code  is  instru¬ 
mented  the  same  way.  For  TCAT/Ada,  the  vendor  recommends  a  capacity  of  up  to  2500 
segments  (approximately  20,000  lines  of  code).  The  vendor  estimates  the  size/perfor¬ 
mance  overhead  for  instrumentation  at  20%  to  30%,  although  this  can  be  higher  for  very 
complex  programs.  For  the  Ada  Lexical  Analyzer  Generator,  TCAT/Ada,  TCAT- 
PATH,  and  S-TCAT/Ada  instrumentation  gave,  respectively,  26%,  27%,  and  14% 
increases  in  code  size.  The  instrumentation  introduced  by  S-TCAT/Ada  can  be  limited 
by  specifying  a  set  of  module  interfaces  that  are  not  to  be  reported  on.  Versions  of  TC  AT 
and  S-TCAT  that  accomplish  various  levels  of  in-place  buffering  to  enhance  performance 
are  available  for  C.  Similar  support  is  not  available  for  the  Ada  versions. 

Ada  restrictions.  The  coverage  analyzers  do  not  instrument  overloaded  routines 
or  tasking  constructs.  The  TCAT/Ada  and  S-TCAT/Ada  processors  (iada  and  s-iada) 
have  been  validated  against  the  Ada  validation  suite,  a  set  of  programs  that  test  compli¬ 
ance  with  the  Ada  standard. 

Problems  encountered.  Several  problems  delayed  the  examination  of  these  tools. 
First  of  ail,  copies  of  testing  tools  for  C  programs  were  sent  twice  before  the  requested 
Ada  versions  finally  arrived.  Except  for  TDGen  and  TSCOPE,  only  the  command-line 
versions  of  tools  were  sent.  The  menu  versions  were  again  requested  but  never  arrived. 
Initial  execution  of  TCAT/Ada  on  one  of  Software  Research’s  example  programs  caused 
a  segmentation  fault  and  sometimes  a  core  dump.  Initial  problems  encountered  with 
TCAT-PATH  and  S-TCAT/Ada  turned  out  to  be  the  result  of  errors  in  the  supplied  nin- 
time  file.  For  all  three  instrumenters,  misplaced  inserted  instrumentation  statements  had 
to  be  manually  corrected.  In  some  cases,  these  misplaced  statements  caused  compilation 
errors.  In  other  cases,  the  incorrectly  instrumented  program  would  compile  but  produced 
the  wrong  results.  Additional  problems  with  accessing  help  were  attributed  to  poor 
installation  procedures  that  caused  help  files  to  be  improperly  positioned.  In  TDGen, 
errors  in  values  and  template  file,  or  in  the  specification  of  program  options,  caused  the 
program  to  hang. 

3.4  TBGEN  and  TCMON 

TBGEN  generates  testbeds  that  facilitate  unit  testing  bottom-up  integration  test¬ 
ing.  The  next  version  of  this  tool  will  include  the  generation  of  stubs  so  that  top-down 
integration  testing  is  also  supported. 
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3.4.1  Tool  Overview 

TBGEN  and  TCMON  are  marketed  by  Nokia  Data  systems.  They  have  been 
commercially  available  since  1986,  although  a  fully  revised  version  of  TBGEN  (Version 
3.0)  was  released  in  May  1989.  30  permanent  multi-user  licenses  have  been  sold. 
Designed  to  be  hardware  architecture,  operating  system,  and  compiler  independent, 
these  tools  are  available  for  VAX/VMS,  Sun-3/SunOS,  PCs  under  MS-DOS  and  OS-2, 
and  Rational  machines.  There  are  some  minor  difference  between  the  versions  available 
on  different  operating  environments;  for  example,  unlike  the  Sun-3/SunOS  versions,  the 
VAX/VMS  tools  do  not  allow  escaping  to  the  operating  system  command  level.  TBGEN 
prices  start  at  $2,850  and  TCMON  at  $2,300.  Evaluation  copies  of  the  tools,  allowing 
their  use  for  60  days,  are  available  for  $1,000  (one  tool)  or  $1,500  (both  tools).  The  ver¬ 
sions  examined  were  TBGEN  Version  3.1  and  TCMON  Version  2.2  operating  on  a 
VAX/VMS  platform. 

Using  Ada  program  unit  specifications,  TBGEN  generates  a  testbed  and  a  com¬ 
mand  file  for  compiling  and  linking  this  testbed  with  the  units  under  test.  The  user  can 
control  the  size  of  a  generated  testbed  by  specifying  particular  subprograms  or  program 
objects  to  be  excluded.  A  log  file  automatically  records  pertinent  information  about 
testbed  generation.  The  user  executes  the  resulting  testbed,  specifying  the  desired  calling 
sequence  and  subprogram  parameters,  and  observing  the  results.  A  powerful  set  of  Ada- 
like  testbed  commands  is  provided.  For  example,  testbed  variables  can  be  declared  and 
their  visibility  directly  controlled,  and  many  of  the  entities  declared  in  Ada  specifications 
can  be  accessed.  Additional  commands  display  information  based  on  current  testbed  set¬ 
tings  and  testbed  status,  or  cause  user  inputs  and  testbed  outputs  to  be  copied  to  a  trace 
file  for  later  examination.  Instead  of  using  a  testbed  interactively,  the  user  can  specify 
testbed  inputs  in  the  form  of  a  script  file.  Scripts  may  be  user  developed  or  generated  in 
the  form  of  a  copy  of  previous  testbed  inputs.  Conditional  and  iteration  control  struc¬ 
tures,  along  with  fixed  and  variable  breakpoints,  are  provide  for  scripts.  Assertions  are 
provided  for  automatic  checking  of  test  results  against  expected  results.  Since  the  testbed 
takes  standard  input  from  the  keyboard  for ’interactive  communication  with  the  user, 
some  difficulties  may  be  encountered  if  a  module  under  test  also  uses  standard  input. 

TCMON  instruments  the  contents  of  user-selected  files  with  statements  that  act 
as  measurement  probes.  In  addition  to  structural  coverage,  these  probes  provide  for  seg¬ 
ment  execution  counts  and  true/false  counts  of  conditions  and  subconditions.  They  also 
provide  timers  that  allow  capturing  execution  times  at  the  program  unit  level,  and  the 
measurement  of  times  between  user-specified  events.  Each  subprogram  can  be  instru¬ 
mented  for  different  types  of  monitoring.  A  test  monitor  is  generated.  A  command  file 
for  compiling  the  monitor  and  instrumented  code  and  performing  necessary  linking  is  also 
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generated,  together  with  a  log  file  providing  information  about  instrumented  files  and 
units  generated.  The  monitor  supports  a  command-driven  interface  that  provides  the 
user  with  commands  to  reset  all  counters  and  timers,  save  and  append  measurement 
data,  produce  a  profile  listing,  run  the  instrumented  program,  etc.  Where  necessary,  this 
interface  can  be  omitted  by  inserting  TCMON  commands  in  source  files  as  special  com¬ 
ments  and  generating  a  dummy  monitor.  Data  generated  by  the  instrumentation  is 
recorded  in  a  profile  listing.  This  gives  detailed  information  about  counter  and  timer 
places  and  values,  and  a  histogram  of  statement  list  execution  counts  is  included.  The 
profile  listing  also  contains  information  that  can  be  used  to  estimate  the  influence  of 
instrumentation  statements  on  measured  time.  The  TCMON  Postprocessor  (TCPOST) 
processes  the  profile  listing  to  generate  summary  reports  at  either  the  package  or  subpro¬ 
gram  level. 

Timers  may  include  invalid  data  when  two  or  more  tasks  call  the  same  instru¬ 
mented  subprogram  or  are  of  the  same  instrumented  task  type.  The  same  is  true  for 
recursive  procedures.  If  this  happens,  the  affected  timers  are  flagged  in  the  profile  listing. 
Although  generic  procedures  and  packages  can  be  instrumented,  multiple  instances  are 
not  distinguished.  Also,  when  returning  from  a  function,  it  is  not  possible  for  a  timer 
within  the  function  to  record  the  time  spent  in  the  evaluation  of  the  return  expression. 
Exceptions,  which  are  invisible  to  the  instrumentor,  may  also  distort  timing  results. 

3.4.2  Observations 

Ease  of  use.  The  user  interacts  with  TBGEN  and  TCMON  through  command 
interfaces  that  are  well  supported  with  prompts  to  guide  a  user  through  necessary  steps. 
Context-sensitive  help  is  available,  together  with  general  descriptions  on  user-selected 
topics.  Error  messages  are  informative,  though  no  specific  help  for  resolving  an  error  is 
provided.  They  are  written  to  both  the  display  and  the  appropriate  log  file.  When  erro¬ 
neous  input  is  detected,  execution  of  the  current  command  is  terminated  and  the  rest  of 
the  current  input  line  ignored.  When  a  test  script  is  being  used  in  TBGEN,  processing  will 
continue  with  the  next  line.  Command  files  are  provided  to  relieve  the  user  of  some 
repetitive  manual  labor.  Although  the  use  of  TCMON  requires  no  special  knowledge,  the 
TBGEN  command-interface  requires  a  knowledge  of  Ada.  All  reports  are  well-struc¬ 
tured  and  clear,  with  useful  history-keeping  information. 

TBGEN  is  tailorable  in  several  ways.  The  SPECIAL  command  implements  envi¬ 
ronment  or  installation  specific  commands.  Configuration  parameters  specified  in  a  sys¬ 
tem  file  can  be  changed,  essentially  to  modify  default  file  names.  A  system  file  gives  the 
specification  for  package  STANDARD  which  can  be  modified  to  reflect  some  of  the 
options  available  to  Ada  compilers.  The  template  files  used  in  generating  testbed 
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components  can  be  changed. 

Some  aspects  of  TCMON  can  also  be  altered  by  modifying  the  template  file  used 
for  generating  auxiliary  Ada  units  and  the  command  file.  This  template  file  also  contains 
the  configuration  parameters  that  can  be  changed  to  alter  default  values.  The  TCMON 
User’s  Manual  provides  suggestions  for  modifying  the  parent  type  for  counter  variables, 
measuring  CPU-time  instead  of  default  elapsed  time,  or  including  other  cost  functions. 

Documentation  and  user  support.  The  documentation  is  well-written  and  guides  a 
user  through  using  each  tool.  The  vendor  provided  good  support  and  answered  all  ques¬ 
tions  quickly  and  well. 

Instrumentation  Overhead.  TCMON  is  designed  to  minimize  the  introduction  of 
unnecessary  instrumentation.  It  not  only  allows  the  user  to  select  the  files  whose  contents 
are  to  be  instrumented,  but  allows  each  file  to  be  instrumented  differently.  TCMON  also 
allows  the  user  to  select  between  SAFE  or  UNCHECKED  modes  for  the  segment 
counter.  The  vendor  cites  a  50%  to  100%  increase  in  code  size  for  full  structural  instru¬ 
mentation.  For  the  Ada  Lexical  Analyzer  Generator,  full  structural  instrumentation  of 
all  units  gave  a  size  increase  of  120%. 

Ada  restrictions.  TBGEN  accepts  any  valid  Ada  code.  Expressions,  however, 
are  skipped  with  the  result  that  the  type  of  an  array  index  cannot  always  be  determined 
automatically  and  the  user  may  be  asked  to  supply  this  information.  Tasks,  task  types, 
and  dependent  entities  are  ignored  and  cannot  be  accessed  in  testbeds  directly.  Similarly, 
testbeds  do  not  provide  the  user  with  access  to  objects  of  limited  type,  functions  with 
results  of  limited  type,  array  objects  with  a  constrained  array  definition,  and  constrained 
subtypes  of  a  type  with  discriminants.  TCMON  may  misinterpret  overloaded  operators 
returning  boolean  values  when  these  are  used  in  conditions. 

Problems  encountered.  No  significant  problems  were  encountered  during  the 
examinations  of  these  tools. 

3.5  TestGen 

TestGen  is  part  of  the  AISLE  family  of  software  tools  that  is  based  on  the  Ada- 
based  Design  and  Documentation  Language  (ADADL).  ADADL  itself  is  fully  compil¬ 
able  by  any  Ada  compiler  and  has  been  selected  by  the  Joint  Integrated  Avionics  Working 
Group  (JIAWG)  as  the  Ada  program  design  language  (PDL)  to  use  for  the  Army’s  Light 
Helicopter  Experimental  (LHX)  and  the  Air  Force’s  Advanced  Tactical  Fighter  (ATF) 
programs.  In  additional  to  the  ADADL  processor  that  provides  static  analysis  of 
designs,  the  tool  family  includes  syntax-directed  and  graphical  editors,  a  design  and  code 
quality  analyzer,  an  automatic  document  generator,  and  design  database  analysis  tools 
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and  supports  requirements  traceability.  It  can  interface  to  the  Teamwork,  Software 
Through  Pictures,  and  Excelerator  CASE  systems  [Hook  91]  to  provide  automatic  gener¬ 
ation  of  designs  from  requirements.  The  examination  focused  on  TestGen  which  can  be 
applied  both  to  ADADL  designs  and  Ada  code. 

3.5.1  Tool  Overview 

The  AISLE  tool  family  is  marketed  by  Software  Systems  Design  (SSD).  It  has 
been  available  since  1984  and  has  over  1,000  users.  The  tools  are  available  on  a  wide 
range  of  machines  such  as  VAX,  VAXStation,  and  Micro  VAX  under  VMS,  UNIX,  or 
Ultrix;  and  Sun-3  and  Sun-4,  HP9000-800,  Apollo,  DecStation,  and  80386-based  PC  sys¬ 
tems  under  MS-DOS  or  UNIX.  Where  windowing  is  required,  the  tools  support  X-Win- 
dows,  SunWindows,  DECwindows,  Tektronix,  and  Hewlett-Packard  windows.  Graphics 
output  is  formatted  for  a  range  of  devices.  These  formats  include  Postscript,  Tektronix, 
and  Graphical  Kernel  System  (GKS).  TestGen  prices  start  at  $4,600  and  those  for 
ADADL,  required  for  TestGen  preprocessing,  at  $5,000.  Training  and  consulting  ser¬ 
vices  are  available. 

These  examinations  used  Demonstration  Version  2.0.3  of  the  tools  operating  on  a 
Sun-4  system.  The  demonstration  version  costs  $150  and  is  supposedly  fully  functional 
with  the  exception  that  input  files  are  limited  to  450  lines.  During  the  examination  of 
TestGen,  however,  it  was  found  that  the  demonstration  version  only  identifies  50%  of 
paths  and  reports  the  coverage  achieved  on  only  1  program  unit.  Additionally,  it  only 
supports  X-Windows  and  SunWindows.  The  restriction  on  input  file  size  precluded  appli¬ 
cation  of  some  parts  of  the  TestGen  on  the  Ada  Lexical  Analyzer  Generator.  In  these 
cases,  SSD  executed  the  necessary  steps  using  a  full  version  of  the  tool  and  returned  the 
results  to  the  examiner  for  analysis. 

Two  parts  of  TestGen  were  considered:  the  Unit  Test  Strategy  Generator  and  the 
Test  Coverage  Analyzer.  (The  third  component,  the  Design  Review  Expert  Assistant  was 
not  examined.)  The  first  of  these  parts  calculates  the  total  number  of  paths  and  branches 
and  the  cyclomatic  complexity  for  each  selected  program  unit.  It  also  identifies  any  unex¬ 
ecutable  paths  and  generates  control  graphs  of  the  code.  This  information  is  used  in 
estimating  testing  costs  in  terms  of  the  number  of  test  cases  required  for  structural  analy¬ 
sis.  TestGen  supports  three  structural  dynamic  analysis  techniques:  path  testing,  branch 
testing,  and  structured  testing  based  on  McCabe’s  cyclomatic  complexity  number 
[McCabe  1976].  Once  the  user  specifies  the  types  of  analysis  required,  the  Unit  Test 
Strategy  Generator  identifies  the  conditions  required  at  each  decision  point  and  the  state¬ 
ments  executed  under  those  conditions.  This  information  helps  the  user  derive  necessary 
test  data  for  structural  testing.  The  Test  Coverage  Analyzer  is  then  used  to  instrument 
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user-specified  program  units.  It  also  generates  a  simple  testbed  for  the  main  Ada  proce¬ 
dure  that  performs  a  loop  calling  the  instrumented  programs  for  as  long  as  the  user 
wishes.  If  a  main  procedure  is  not  present,  a  special  testbed  must  be  manually  created  by 
the  user  based  on  a  template  supplied  in  the  documentation.  The  user  compiles  and  links 
the  instrumented  code  and  testbed.  As  the  program  executes  on  test  data,  the  instrumen¬ 
tation  produces  a  trace  history  that  records  the  order  in  which  statements  were  executed. 
The  Test  Coverage  Analyzer  is  then  used  to  analyze  this  trace  history  and  report  on  state¬ 
ment  and  path  coverage,  and  statement  and  program  unit  execution  profiles.  Reporting 
on  the  coverage  accumulated  over  a  series  of  test  runs  is  achieved  by  requiring  analysis  of 
multiple  trace  histories. 

3.5.2  Observations 

Ease  of  use.  TestGen  is  menu-driven,  requiring  input  from  the  keyboard.  An 
on-line  manual  is  provided  instead  of  on-line  help.  Error  messages  are  terse  and  only 
minimal  checking  of  user  input  is  provided.  For  example,  in  one  case  the  lack  of  check¬ 
ing  for  invalid  file  names  led  to  a  segmentation  fault.  No  special  knowledge  is  required  to 
use  the  tool.  All  output  reports  are  well-structured  and  provide  easy-to-read  information. 
A  major  strength  of  this  tool  is  the  clear  identification  of  the  path  conditions  that  guide 
the  execution  of  particular  program  paths.  Other  than  setting  default  values  for  files 
names,  the  user  interface  cannot  be  tailored. 

Documentation  and  user  support.  The  installation  instructions  were  not  very  clear, 
but  other  documentation  was  good.  SSD  staff  provided  quick  and  helpful  support.  They 
acted  on  problem  reports  immediately,  providing  resolutions  to  the  encountered  problems 
within  a  day  or  even  within  a  couple  of  hours. 

Instrumentation  Overhead.  The  entire  program  must  be  analyzed  by  the  ADADL 
processor  before  the  Unit  Test  Strategy  Generator  can  be  used.  Subsequently,  TestGen 
functions  can  be  invoked  for  the  files  analyzed  by  the  ADADL  processor.  The  size  of 
instrumented  code  is  minimized  by  allowing  the  user  to  specify  which  modules  in  the 
selected  file  should  be  instrumented.  All  selected  modules  are  instrumented  in  the  same 
fashion.  Instrumentation  of  the  Ada  Lexical  Analyzer  Generator  gave  a  50%  increase  in 
code  size.  TestGen  accumulates  the  data  generated  by  instrumentation  statements  in 
memory;  this  may  limit  the  amount  of  code  that  can  be  monitored. 

Ada  restrictions.  The  Unit  Test  Strategy  Generator  can  analyze  any  Ada  code, 
but  the  Test  Coverage  Analyzer  cannot  instrument  tasks.  Source  lines  with  multiple  state¬ 
ments  may  be  instrumented  incorrectly. 
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Problems  encountered.  TestGen  requires  that  the  software  under  test  first  be  ana¬ 
lyzed  by  the  ADADL  processor;  problems  were  encountered  during  the  initial  use  of  this 
processor.  In  one  case,  the  ADADL  processor  went  into  an  infinite  loop  when  run  on 
example  code  provided  by  SSD.  These  problems  were  reported  to  SSD  who  subsequently 
identified  one  defect  in  the  ADADL  processor  and  two  defects  in  TestGen.  Workarounds 
were  provided.  Some  problems  were  the  result  of  inadequate  documentation  or  error 
checking.  For  example,  TestGen  does  not  check  whether  the  specified  window  format  is 
actually  available;  if  it  is  not,  the  tool  crashes.  It  should  be  remembered  that  the  exami¬ 
nation  was  performed  on  the  demonstration  version  of  TestGen;  these  problems  may  not 
occur  with  the  licensed  versions. 
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4.  CONCLUSIONS 


An  overview  of  the  examined  tools  is  provided  by  Table  3  which  summarizes  the 
tool  features  previously  reported.  The  first  part  of  this  section  lists  findings  concerning 
the  status  of  the  examined  tools  that  may  affect  their  use.  The  last  part  comments  on 
identified  gaps  in  testing  tool  functionality. 

4.1  Findings  on  the  Status  of  Commercial  Tools 

Findings  drawn  from  the  examination  of  these  commercial  tools  are  as  follows: 

Finding  1:  Some  of  the  examined  tools  could  be  brought  into  immediate  use  to 
improve  the  cost-effectiveness  of  testing  for  SDI  software  development  efforts.  The 
examined  tools  provide  various  ways  to  improve  the  consistency  of  defect  detec¬ 
tion  and  reduce  testing  costs.  Those  that  were  found  to  be  robust  and  easy  to  use 
are  suitable  for  immediate  use.  In  particular,  TBGEN  offers  valuable  support  for 
automatically  generating  the  test  beds  needed  for  both  unit  and  integration  testing. 
TCMON  not  only  provides  for  ensuring  the  adequacy  of  a  set  of  test  data  in  terms 
of  the  level  of  structural  coverage  required,  but  is  the  only  examined  tool  that  sup¬ 
ports  timing  analysis.  TBGEN  and  TCMON  also  provide  useful  records  of  testing 
activities.  SoftTest  supports  the  systematic  development  of  test  data  and  measures 
test  data  adequacy  in  terms  of  functional  coverage.  TDGen  provides  systematic 
generation  of  large  amounts  of  user-defined  test  data. 

Finding  2:  The  coverage  analyzers  provide  reporting  data  that  can  support  the  man¬ 
agement  of  SDI  testing  efforts.  Several  of  the  examined  tools  support  test  manage¬ 
ment  by  reporting  on  test  progress  in  terms  of  percentage  of  the  required  structural 
coverage  that  has  been  achieved  to  date.  The  majority  of  these  tools  are  intended 
to  measure  coverage  at  the  unit  testing  but  base  their  measurement  on  different 
structural  features.  Although  these  tools  provide  module  execution  coimts  that 
provide  some  support  for  measurement  at  the  integration  level,  only  S-TCAT/Ada 
measures  coverage  achieved  during  integration  testing.  The  coverage  analyzers 
vary  in  their  ability  to  estimate  the  number  of  test  cases  needed,  to  support  the 
development  of  test  data,  and  to  handle  archive  files  of  test  results.  Only  TCMON 
allows  the  user  to  specify  the  required  coverage  level. 
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TABLE  3.  Summary  of  Tool  Features 


Tool  Name 
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O 

O 

O 

O 

• 

• 
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• 

• 

• 

o 

- 

- 

• 
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© 
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© 
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o 
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O 
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On-Line  Help 

- 
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o 
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• 

• 

- 

© 

- 
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o 
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o 
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• 

O 

• 

© 

Test  Phases 

U4 

s 

U4,S  U4 

I 

u 

u 

uj 

u 

u 

Activity  Logging 

O 

o 

- 

0 

• 

• 

- 

- 

© 

- 
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- 

- 

- 

- 

- 

- 

teat 
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- 
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• 

n/a 
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O 

• 

• 

• 
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© 

• 

Key  to  Related  Products: 
s  -  seminars/consulting 
t  -  testing  services 
V  -  language  validation  suites 


O  ©  •  •  • 

Better  Worse 


Key  to  Test  Phases  Supported: 
U  -  unit  testing 
I  ‘  integration  testing 
S  -  system  testing 
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Finding  3:  Many  of  the  tools  can  be  used  in  coiyunction  to  overcome  the  limitations 
of  particular  tools.  The  tools  are  all  intended  to  provide  support  for  specific,  lim¬ 
ited  testing  activities.  No  single  tool  provides  all  desirable  functionality.  MAL- 
PAS,  SoftTest,  S-TCAT/Ada,  TBGEN,  and  TDGen  each  provide  different  func¬ 
tionality.  They  all  operate  in  an  independent  manner  and  can  be  used  in  conjunc¬ 
tion  with  each  other.  TCAT/Ada,  TCAT-PATH,  TCMON,  and  TestGen  provide 
similar  types  of  functionality;  each  can  be  used  with  any  combination  of  MAL- 
PAS,  SoftTest,  and  TDGen.  In  addition  to  its  companion  tool  TCMON,  it  may  be 
possible  to  use  TBGEN  to  generate  test  beds  for  use  with  TCAT/Ada  and  TCAT- 
PATH. 

Finding  4:  Only  a  few  of  the  tools  support  test  data  generation,  and  this  support  is 
generally  indirect.  Only  TDGen  actually  generates  test  data.  This  tool  operates 
independently  of  a  functional  specification,  design,  or  code.  It  is  independent  of 
any  particular  test  strategy  and  the  effectiveness  of  generated  test  data  depends  on 
the  user.  Some  other  tools,  namely  SoftTest  and  TestGen,  identify  values  that  cer¬ 
tain  program  variables  should  take,  requiring  a  user  to  work  backwards  to  deter¬ 
mine  the  necessary  values  for  program  inputs. 

Finding  5:  The  use  of  some  tools  may  impose  restrictions  on  code  development. 
Only  SoftTest  and  TDGen  are  independent  of  code.  The  other  tools  are  all  subject 
to  restrictions  on  the  Ada  code  they  can  process.  Additionally,  TBGEN  and  all 
the  coverage  analyzers  receive  data  on  the  standard  input  stream.  This  presents 
present  difficulties  when  testing  programs  that  also  use  the  standard  input  stream. 
TCMON  is  the  only  tool  that  provides  a  workaround  for  this  problem.  Additional 
limitations  may  be  found  with  further  use  of  the  tools. 

Finding  6:  The  overhead  incurred  by  code  instrumentation  needed  for  structural 
coverage  analysis  can  be  a  significant  factor.  The  examined  coverage  analyzers  all 
limit  unnecessary  increases  in  program  compilation  and  execution  times  by  allow¬ 
ing  the  user  to  specify  files  whose  contents  are  to  be  instrumented.  Only  TestGen 
goes  beyond  this  and  allows  the  explicit  specification  of  the  particular  programs 
units  to  instrument.  How  a  tool  gathers  the  large  amount  of  data  that  instrumenta¬ 
tion  typically  generates  is  also  important.  The  amount  of  code  that  can  be  moni¬ 
tored  may  be  restricted  if  this  data  is  accumulated  in  memory.  The  alternative 
approach,  writing  data  to  a  file,  incurs  a  performance  overhead. 

Finding  7:  The  tools  do  not  require  special  skills  but  some  are  immature.  The  tools 
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require  little  sophistication  on  the  part  of  the  user  and  are  supported  by  good  docu¬ 
mentation.  Some  actively  guide  a  user  through  necessary  tasks,  keep  a  record  of 
test  activities,  and  take  extra  steps  to  relieve  the  user  of  repetitive  tasks.  In  gen¬ 
eral,  however,  the  tools  employ  primitive  user  interfaces  that  could  benefit  from 
the  application  of  human  factors  engineering.  In  two  cases,  MALPAS  and  TCAT- 
PATH,  the  need  to  refer  to  separate  listings  to  identify  objects  referenced  in 
reports  complicated  tool  use.  There  were  instances  where  different  tools  gave  dif¬ 
ferent  results  when  performing  the  same  function,  for  example,  calculating 
cyclomatic  complexity.  Moreover,  some  of  the  tools  contained  faults.  While  most 
defects  were  trivial,  others  rendered  a  tool  unusable  until  fixed  by  the  vendor.  In 
three  cases,  TCAT/Ada,  TCAT-PATH,  and  TestGen,  major  failures  occurred 
when  using  the  tool  on  sample  software  supplied  by  the  vendor. 

Finding  8:  The  examined  tools  all  support  mature  testing  approaches.  The  func¬ 
tional  and  structural  testing  approaches  supported  by  the  majority  of  these  tools 
have  been  available  since  the  early  1970s.  Although  the  effectiveness  of  these 
approaches  cannot  be  directly  stated  in  terms  of  freedom  from  defects,  there  is 
considerable  empirical  evidence  of  their  benefits  [Rowland  1989,  Duran  1984, 
Girgis  1986,  Selby  1986].  This  is  not  to  say  that  they  carmot  be  improved;  for 
example,  several  test  strategies  have  been  developed  that  offer  a  compromise 
between  the  high  cost  of  path  testing  and  the  lack  of  stringency  of  branch  testing 
[Woodward  1980,  Harrold  1989].  These  alternative  strategies  could  be  useful 
employed  in  coverage  analyzers. 


More  data  on  the  practiced  costs  and  benefits  of  particular  tools  is  needed  to  determine 
which  tools  should  be  recommended  for  use,  to  justify  these  recommendations,  and  to 
encourage  prospective  users.  It  could  also  be  used  to  determine  further  necessary  prod¬ 
uctization  efforts  or  the  porting  of  tools  to  new  operating  platforms.  To  this  end,  SDI  soft¬ 
ware  development  managers  should  be  encouraged  to  conduct  their  own  examinations  of 
testing  tools,  perhaps  using  this  report  as  an  initial  guideline.  Lessons  learned  in  the 
course  of  such  efforts  should  be  fed  back  to  the  tool  developers  to  guide  the  improvement 
of  existing  tools  and  the  development  of  new  tools. 

4.2  Functional  Gaps  in  Conunercial  Testing  Tools 

Despite  the  efforts  of  several  researchers,  for  examples  see  [Goodenough  1975, 
Cherniavsky  1988,  Morell  1988],  there  is  no  commonly  agreed  formal  theory  of  software 
testing.  In  the  absence  of  such  a  theory,  a  definitive  specification  of  the  required  func¬ 
tionality  of  testing  tools  carmot  be  developed.  One  alternative  way  of  compiling  a  list  of 


desired  functionality  is  to  look  at  the  new  testing  techniques  that  researchers  are  investi¬ 
gating.  This  type  of  approach  could  result  in  a  list  that  requires  automated  support  for  a 
large  number  of  techniques  such  as  assertion  testing  [Luckham  1986],  data  flow  testing 
[Frankl  1986],  flavor  analysis  [Howden  1987],  mutation  testing  [DeMillo  1988],  and  soft¬ 
ware  fault  tree  analysis  [Cha  1988]. 

A  more  pragmatic  approach  is  to  look  at  testing  practices  and  identify  where 
automated  support  is  needed  to  alleviate  difficulties  in  current  practices.  This  is  the 
approach  adopted  here.  Based  on  an  assessment  of  these  difficulties,  and  the  capabilities 
of  the  set  of  testing  tools  identified  at  the  start  of  this  work,  some  of  the  most  critical  gaps 
in  commercial  tool  functionality  are  the  following: 

•  Support  for  reproducible  testing  of  concurrent  software.  The  inherent  indeter¬ 
minism  of  concurrent  programs  severely  complicates  path  analysis  and  means 
that  two  executions  of  the  same  software  with  the  same  inputs  can  produce  differ¬ 
ent  behaviors.  This  lack  of  reproducibility  handicaps,  for  example,  determining 
the  cause  of  a  failure  and  retesting  a  modified  program.  Concurrent  programs 
can  also  contain  a  new  class  of  faults,  called  synchronization  faults.  Additional 
tests  are  needed  to  check  for  existence  of  these  faults. 

•  Automated  oracles  that,  for  stated  test  inputs,  can  determine  the  expected  out¬ 
puts.  In  dynamic  testing,  determining  the  expected  program  outputs  for  each  set 
of  test  data  is  equally  important  to  generating  the  test  data  itself.  It  is  also  more 
difficult  and  time  consuming.  Since  technology  for  automated  oracles  is  not,  and 
may  never  be,  available,  the  development  of  expected  outputs  will  continue  to  be 
a  major  cost  source.  A  number  of  alternative  approaches  have  been  investigated. 
These  include  testing  techniques  that  avoid  the  need  for  an  oracle  [DeMillo  1988] 
and  the  use  of  N-version  programming  [Avizienis  1985].  The  use  of  symbolic  eval¬ 
uation  for  providing  a  partial  oracle  is  perhaps  the  most  promising  approach 
[Richardson  1985]. 

•  Automated  assessment  of  software  reliability.  Currently,  software  reliability 
models  are  the  only  way  of  assessing  software  reliability.  Although  the  Naval  Sur¬ 
face  Weapons  Center  has  developed  a  tool  that  automates  some  of  the  most  pop¬ 
ular  models,  there  is  a  need  for  supported  commercial  tools.  Meanwhile,  these 
models  are  based  on  hardware  reliability  theory  and  their  validity  in  the  software 
arena  has  been  questioned.  Further  understanding  of  software  reliability  is 
needed. 

Research  is  being  conducted  in  these  areas.  As  appropriate,  the  timely  development  of 
prototypes  should  be  encouraged  to  help  this  technology  to  mature. 
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5.  FURTHER  WORK 


In  determining  the  current  scope  of  the  study,  the  examination  of  several  tools  or 
tool  classes  was  postponed.  Examination  of  these  tools  and  tools  classes  will  provide  a 
better  understanding  of  the  available  commercial  tools  that  support  testing  of  Ada  code. 
This  section  identifies  specific  tools  and  tool  classes  that  should  be  included  to  complete 
this  study. 

5.1  Additional  Static  and  Dynamic  Analysis  Tools 

During  the  initial  data  gathering  activities  of  this  work  several  promising  static 
and  dynamic  analysis  tools  were  identified  that,  because  of  limited  resources  or  other  rea¬ 
sons,  were  not  included  in  the  examinations.  As  previously  discussed,  examination  of  the 
T  testing  tool  has  been  postponed  pending  the  availability  of  a  significantly  revised  ver¬ 
sion  due  for  release  in  September  1991.  The  first  complete  version  of  AdaQuest  is  still 
awaited.  Preliminary  information  on  AdaQuest  and  T  is  given  below.  Additional  analy¬ 
sis  tools  that  should  be  examined  include  the  following: 

•  Static  and  Dynamic  Code  Analyzer  (ARC  SADC A)  from  Optimization  Technol¬ 
ogy,  Inc.  The  Ada  version  of  this  new  tool  is  expected  by  the  end  of  1991. 

•  TST  from  Intermetrics.  A  testbed  generation  tool  for  Ada  subprograms  and 
tasks,  this  tool  is  based  on  the  Ada  Test  Support  Tool  whose  development  was 
funded  by  the  STARS  program.  Its  commercial  availability  is  still  uncertain. 

•  Analysis  of  Complexity  Tool  (ACT),  Battlemap  Analysis  Tool  (BAT),  Design 
Complexity  Tool,  and  Structure  Testing  and  Requirements  Tool  (START)  from 
McCabe  and  Associates.  These  tools  provide  complexity  analysis,  test  path  and 
.ondition  generation,  integration  test  generation,  and  data  flow  scenario  genera¬ 
tion.  The  Ada  version  of  the  new  Instrumentation  Tool  is  expected  to  become 
available  by  the  end  of  1991. 

5.1.1  Preliminary  Information  on  AdaQuest 

AdaQuest  is  under  development  for  operation  on  VAX  machines.  Advertised 
prices  depend  on  the  operating  environment  and  start  at  $6,500.  The  dynamic  analyzer 
supports  branch  testing  and  verifies  unit  and  subsystem  timing  requirements  by  measuring 
the  actual  time  spent  in  user-specified  code  sections.  Assertions  are  used  for  checking 
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unit-internal  design  constraints  and  interface  constraints.  In  addition  to  supporting  test 
data  preparation  for  branch  testing,  the  static  analyzer  checks  for  logic  errors  such  as  infi¬ 
nite  loops,  unreachable  statements,  and  uninitialized  variables.  It  also  checks  for  viola¬ 
tion  of  project-specific  coding  standards  and  generates  various  dependency  reports. 
Finally,  the  task  analyzer  traces  the  actual  synchronization  relationships  between  Ada 
tasks,  creating  timing  diagrams  that  may  help  in  diagnosing  synchronization  errors  such 
as  deadlock  and  starvation. 

5.1.2  Preliminary  Information  on  T 

T  is  already  used  by  various  government  organizations,  including  the  Naval 
Avionics  Center,  the  Jet  Propulsion  Laboratories,  Naval  Coastal  Systems  Center,  and  the 
US  Army  Forts  Monmouth  and  Sill.  It  is  available  for  PC/MS-DOS  and  VAX/VMS  plat¬ 
forms,  and  various  workstations  under  UNIX.  Training  is  a  prerequisite  for  tool  purchase 
and  costs  $1,500  per  person  at  PEI,  or  $10,000  for  an  on-site  workshop.  Software  prices 
start  at  $7,000.  An  interface  between  T  and  the  Teamwork  CASE  system  is  due  to  be 
marketed  by  Cadre  Technologies,  Inc.  in  early  1992. 

T  generates  test  cases  from  requirements  information.  This  information  includes 
details  on  software  actions,  data,  events,  conditions,  and  states  and  is  specified  in  the  T 
Software  Description  Language  (TSDL).  T’s  goal  is  generation  of  the  minimum  number 
of  traceable  test  cases  that  will  exercise  every  operation  and  each  of  a  set  of  vendor- 
defined  probable  errors  at  least  once.  It  can  be  used  during  any  software  development 
stage;  during  maintenance  test  cases  are  generated  for  software  changes  only.  In  addition 
to  test  data  generation,  it  provides  tracing  between  tests  and  defined  software  actions. 
From  user-specified  pass/fail  results,  T  also  provides  a  measure  of  test  adequacy  based 
on  requirements  coverage,  input  domain  coverage,  output  range  coverage  and,  option¬ 
ally,  structure  coverage. 

5.2  Other  Classes  of  Testing  Tools 

Regression  testing  tools  are  widely  believed  to  offer  significant  cost  savings  during 
software  development  and  maintenance.  For  example,  based  on  its  industrial  research 
and  testing  services,  one  leading  testing  company  reports  that  30%  of  a  maintenance  bud¬ 
get  is  typically  spent  on  retesting  software,  and  automated  regression  testing  can  result  in 
savings  of  20-25%  [SR  1988b].  A  sample  set  of  regression  testing  tools  should  be  exam¬ 
ined. 


There  is  increasing  evidence  that  CASE  systems  are  starting  to  support  testing 
activities.  A  recent  survey  of  CASE  vendors,  performed  on  behalf  of  the  US  Air  Force, 
found  that  nearly  25%  of  the  examined  products  claim  explicit  support  for  software 
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testing  activities  [Hook  91].  The  goal  of  integrating  testing  support  into  a  CASE  system  is 
to  provide  easy  access  to  testing  tools  that  facilitates  continual  evaluation  of  evolving  soft¬ 
ware.  This  evaluation  can  be  used  to  ensure  timely  detection  of  faults  and  provide  the 
software  developer  with  feedback  to  guide  the  development  process.  The  tools  examined 
in  this  report  can  be  used  in  conjunction  with  any  CASE  system  that  supports  the  develop¬ 
ment  of  Ada  software.  Integration  into  a  CASE  system,  however,  requires  representing 
the  testing  processes  and  products  in  the  supported  life  cycle  model.  A  sample  set  of 
those  CASE  systems  that  offer  testing  support  should  be  examined  to  determine  how 
thorough  and  well-integrated  this  support  is. 
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APPENDIX  A 


VENDORS  OF  TESTING  TOOLS 

TABLE  A-1.  Vendors  of  Commercially  Available  Testing  Tools 


Vendor 

Phone  # 

Tool  Name  and  Function 

Lang. 

ABRAXAS  Software  Inc. 

(800)347-5214 

CODE  CHECK  complexity,  portability,  style  analyzer 

C,  C++ 

AETECH,  Inc. 

(619)755-1277 

AdaScope  interfaces  symbolic  debugger  to  code 

Ada 

ACT  Corp. 

(212)696-5600 

AdaSoft  analyzer/debugger 

Ada 

ASA  Inc. 

(214)245^553 

Hindsight  logic,  complexity,  performance,  productivity 
analyzer 

C 

ATI 

(212)354-8280 

superCASE  CASE  system  w*  dynamic  analyzer 

Fortran 

Alsys  Inc. 

(617)2704)030 

AdaTune  non-intrusive  performance,  coverage  analyzer 

Ada 

BTree  Verification  Systems,  Inc. 

(612)474-3756 

SVS  regression  analyzer 

n/a 

Bender  and  Associates 

(415)924-91% 

SoftTest  functional  test  data  generator 

n/a 

Cadre 

(703)  875-8670 

SAW  non-intrusive  performance  analyzer 

Ada,  C 

XRAY  code  execution  simluator/debugger 

CodeMap  coverage,  stack  space  analyzer 

SmartProbe  debugger  w’  trace  and  breakpoints 

Evaluator  regression  analyzer 

n/a 

Charles  Stark  Draper  Lab. 

DARTS  CASE  system  w*  design  analyzer 

n/a 

Clyde  Digital  Systems,  Inc. 

(801)224-5306 

CARBONCopy  regression  analyxer 

n/a 

Computer  Associates 

(203)627-8923 

TRAPS  regression,  acceptance  tester 

n/a 

Concurrent  Computer  Corp. 

(201)758-7531 

XPA  performance  analyzer 

n/a 

Simulation  Wbench  real-time  performance  analyzer 

n/a 
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Vendor 

Phone  # 

Tool  Name  and  Function 

Lang. 

Convex  Computer  Corp. 

(214)497-4383 

CXpa  performance  analyzer 

n/a 

DDC-1 

(602)  944-1883 

DACS  compiler,  non-intrusive  symbolic  debugger 

Ada 

Digital  Equipment  Corp. 

(800)332-4636 

PCA  coverage,  performance  analyzer 

n/a 

SCA  static  analyzer 

n/a 

Test  Manager  regression,  test  analyzer 

n/a 

Direct  Technology 

(800)  992-9979 

Automator  qa  regression  and  test  manager 

n/a 

Donatech  Corp. 

(515)472-7474 

Realliffle  Testware  non-intrusive  executor  w’  test  manage¬ 
ment 

n/a 

Dynamic  Research  Corp. 

(508)475-9090 

AdaMAT  quality  analyzer 

Ada 

EVB  Software  Engineering  Inc. 

(301)695-6960 

D  YN  complexity  analyzer 

Ada 

General  Research  Corp. 

(805)  964-7724 

AdaQuest  static,  dynamic  analyzer 

Ada 

J73AVS  static,  dynamic  analyzers 

Jovied 

RXVP80  coverage,  static  analyzer 

Fortran 

Gimpel  Software 

(215)584-4261 

Generic  Lint  static  analyzer 

C 

Coldbrick  Software 

(714)760-9196 

Bloodhound  regression  analyzer 

n/a 

Harris  Corp. 

(305)  977-5573 

AMS  quality  analyzer 

Ada 

Hewlett  Packard 

(301)670-4300 

HP  Branch  Validator  coverage  analyzer 

C,  C++ 

m  Avionics 

(201)284-5030 

UATL  driver  for  test  execution 

Ada 

i'Logix 

(617)  272-8090 

Stalemate  CASE  system  w*  testing  functions 

Ada,C 

Information  Processing  Tech.  Inc. 

(415)494-7500 

FORTRAN-lint  static  analyzer 

Fortran 

lint-PLUS  static  analyzer 

C 

Inst,  for  Information  Industry 

•^886  2  7377187 

KangaTool/CEA  cause-effect  analyzer 

n/a 

KangaTool/DPA  dynamic  program  analyzer 

various 

KangaTooi/SPA  static  program  analyzer 

KangaTool/UTT  unit  test  tool 

various 

various 

various 


Vendor 


Phone  # 


Tool  Name  and  Function 


Lang 


Integrated  Systems,  Inc. 

(408)  980-1500 

Autocode  CASE  system  w*  testing  functions 

various 

Intermetrics,  Inc. 

(714)  891-4631 

TST  Symbolic  debugger,  test  data  generator;  path,  perfor¬ 
mance,  coverage  analyzer 

Ada 

JADE 

(804)  744-5849 

JADE  animator,  monitor,  debugger  for  simulation 

n/a 

Jackson  Systems  Corp. 

(203)  683-1976 

APJ  Workbench  Animator  for  test  data  against  data/program 
structures 

Ada 

MCC 

(512)  338-3345 

SMDC  quality  analyzer 

Ada 

McCabe  and  Associates 

(301)  596-3080 

ACT  complexity,  test  paths/conditions  analyzer 

various 

BAT  test  path/condition  generator 

various 

DCT  design  complexity,  integ.  test 

various 

START  data  flow  scenario  generator 

n/a 

Mentor  Graphics,  Inc. 

(714)  660-8080 

CASE  Station  CASE  system  w'  testing  functions 

C 

Mercury  Interactive  Corp. 

(408)  982-0100 

Runner  series  regression  tester 

n/a 

Microtec  Research 

Xray/DX  performance,  coverage  analyzer  for  embedded  code 

n/a 

Qual  Trak  Corp. 

(408)  274-8867 

DDTs  defect  tracking  system 

n/a 

fDDTs  field  defect  tracking  system 

n/a 

Nokia  Data 

+358-31-237317 

TBGEN  test  bed  generator 

Ada 

TCMON  coverage,  performance  analyzer 

Ada 

Pacific-Sierra  Research 

(213)  820-2200 

Flint  static  analyzer 

Fortran 

Performance  Awareness 

(919)  870-8800 

preVue  series  functional,  regression,  performance  tester 

n/a 

Performance  Software 

(800)  873-6587 

V-Test  regression  analyzer 

n/a 

Pilot  Research  Associates,  Inc. 

(703)883-2522 

Check*Mate  regression  analyzer 

n/a 

Popkin  Software 

(212)  571-3434 

System  Architect  CASE  system  w*  testing  functions 

n/a 

Programming  Environments  Inc. 

(201)  918-0110 

T  functional  test  case  generator 

n/a 

Programming  Research 

(817)  5894)949 

PR;QA  quality  analyzer 

C 
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Vendor 

Phone  # 

Tool  Name  and  Function 

Lang. 

RTF  Software  Ltd. 

+0252  711414 

MALPAS  static  analyzer 

various 

RJO 

(301)731-3600 

Auto-G  CASE  system  w’  testing  functions 

n/a 

Reasoning  Systems 

(415)  494-6201 

Refine  CASE  system  w*  testing  functions 

C 

Howard  Rubin  Associates,  Inc. 

(914)764-4931 

RA-METRICS  project  management  analyzer 

FPXpert  complexity  analyzer 

n/a 

n/a 

SQL  Systems  International 

+44  279-641021 

PCMS’ADA  dependency  analyzer 

Ada 

Set  Labs 

(503)  289-4758 

UX-METRIC  quality  analyzer 

PC-METRIC  quality  analyzer 

various 

various 

Softool  Corporation 

(805)  683-5777 

CAC/2167A  DoD-STD-2167A  compliance  analyzer 

Scandura  Intelligent  Systems 

(215)  664-1207 

PRODOC  CASE  system  w’  coverage  analyzer 

Science  Applications  Inter.  Corp. 

(703)  553-6171 

MAT  maintenance  analysis  tool 

Path  Analysis  Tool  coverage  analyzer 

Fortran 

Fortran 

Software  Quality  Automation 

(800)  228-9922 

SQAManager  test  planner  w*  test  management 

n/a 

Software  Recording,  Inc. 

(214)368-1196 

AutoTester  test  planner  w*  regression  testing 

n/a 

Software  Research  Inc. 

(415)957-1441 

TCAT  branch  coverage  analyzer 

S-TCAT  function  call  coverage  analyzer 

TCAT-PATH  path  coverage  analyzer 

T-SCOPE  coverage  animator 

TDGen  random  and  sequential  test  data  generator 

SMARTS  regression  tester 

CAPBAK  keystroke  Capture  and  Playback  System 

EXDIFF  file  comparator 

various 

various 

various 

n/a 

n/a 

n/a 

n/a 

n/a 

Software  Systems  Design,  Inc. 

(714)625-6147 

TestGen  complexity,  path,  coverage  analyzer 

QualGen  design,  code  quality  analyzer 

AIEM  database  design  analyzer 

Ada,C 

Ada,C 

Sterling  Software 

(818)716-1616 

TestPro  test  planner  w*  test  management 

n/a 
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Vendor 

Phone  # 

Tool  Name  and  Function 

Lang. 

Syscon  Corporation 

(619)296-0085 

PEP  performance  analyzer 

GRAP  animator  w*  CPU  usage,  call  frequency  analyzer 

System  Design  and  Development 

(303)  449-3634 

DCATS  regression  analyzer 

Tartan  Laboratories,  Inc. 

(412)  856-3600 

Ada  Scope  analyzer 

Ada 

Teledyne  Brown  Engineering 

352-8533 

ACAT  complexity  analyzer 

Ada 

SMART  quality  assurance  system 

Ada 

TAGS/RT  CASE  system  w*  testing  support 

n/a 

TeleSoft 

(619)  457-2700 

Telegen2  development  environ’^ent  w*  profiler 

Ada 

Tiburon  Systems,  Inc. 

(703)  920-2321 

FERRET  testing  workstation 

various 

Verilog  SA 

(703)  354-0371 

Logiscope  complexity,  required  tests,  coverage  analyzer 

various 

AGE  CASE  system  w’  r/qs  test  case  generator 

n/a 

XA  Systems  Corp. 

(800)  344-9223 

PATHVU  quality  analyzer 

Cobol 

Xinotech  Research 

(612)379-3844 

Program  Composer  analyzer 

Ada 
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APPENDIX  B 


SAMPLE  OUTPUTS  FROM  MALPAS 


Due  to  MALPAS’s  restrictions  on  analysis  of  Ada  access  types,  the  lexical 
analyzer  code  used  as  a  sample  test  program  could  not  be  thoroughly  analyzed.  To  illus¬ 
trate  the  reports  that  MALPAS  produces  a  simple  Pascal  program  was  substituted.  This 
program  and  the  MALPAS  analysis  reports  are  shown  in  the  following  figures. 
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program  average  (  input ,  output  ) ; 

{  This  program  shares  a  stream  between  two  consumers  by  merging  the 
[  processes  and  evaluating  the  result  of  the  second  process  eagerly. 

{  consumer  process  result  type  ) 

{  stream  element  type  ] 

{  result  returned  by  consumer  #1  ) 

{  result  returned  by  consumer  #2  ) 


type 


var 


resulttype  =  integer; 
streamelement  =  integer; 
conslresult:  resulttype; 
cons2result:  resulttype; 


1 

) 


{  Strezun  operations  } 

procedure  advance  (var  eos:  Boolean;  var  next:  streamelement;  more:  Boolean) 
const  CR  =13;  {  Advance  the  actual  input  stream  ) 

var  ch :  char ; 

begin 

if  more  then 
if  eof  then 
eos  :=  true 
else  begin 

eos  :=  false; 
if  eoln  then  begin 
readln; 
next  :=  CR 
end 

else  begin 
read ( ch ) ; 
next  :=  ord(ch) 
end 

end 

end; 

procedure  consume;  {  Consume  the  input  stream  as  one  process  ) 

var  eos:  Boolean;  (  (count  streeun  elements  and  sum  stream  elements)  ) 
next:  streeunelement; 

begin 

conslresult  :=  0; 
cons2result  :=  0; 
advance ( eos , next , true ) ; 
while  not  eos  do  begin 

conslresult  :=  conslresult  +  1;  {  count  stream  elements  ) 

cons2result  :=  cons2result  +  next;  {  sum  stream  elements  ) 

advance ( eos , next , true ) 
end; 

end; 


begin 

consume; 

writeln('The  average  of  conslresult:!, 

'  characters  is  chr(cons2result  div  conslresult),  '".') 

end. 


Figure  B-1.  Sample  Pascal  Code  Dlustrating  MALPAS  -\nalyses 


TITLE  average; 


[  Pascal  to  Malpas  IL  Translator  -  Release  3.0  ] 


"The  average  of 
"  characters  is  " "" 


[6]  _INCLUDE/NOLIST  "USR : [ADATEST . PASCALIL30] FIXED . PREAMBLE" 
***  Including  file  USR: [ADATEST. PASCALIL30]FIXED. PREAMBLE,!  *** 

***  End  of  file  USR: [ADATEST. PASCALIL30] FIXED. PREAMBLE,!  *** 

[7]  _INCLUDE/NOLIST  "USR: [ADATEST . PASCALIL30] TEXT . PREAMBLE" 
***  Including  file  USR: [ADATEST. PASCALIL30] TEXT. PREAMBLE,!  *** 

***  End  of  file  USR: [ADATEST. PASCALIL30] TEXT. PREAMBLE;!  *** 

[8] 

[!0]  CONST  cr  :  integ.’ir  =  +!3; 

[!!]  CONST  lit _ ! _ theaverage  :  char-array  =  "The  average  of  ' 

[12]  CONST  lit _ 2 _ characters  :  char-array  =  "  characters  is  ' 

[13]  CONST  lit _ 3  :  char-array  = 

[14]  [•  result  returned  by  consumer  #2  *] 

[16] 

[17]  [*  Stream  operations  *] 

[!8] 

[20]  PROCSPEC  advance ( INOUT  eos  :  boolean, 

[21]  INOUT  next  ;  integer, 

[22]  IN  more  :  boolean) 

[23]  IMPLICIT  ( [**  IL  Global  Pareuneter  Section  **  ] 

[24]  INOUT  input  :  text); 

***  WARNING  :  no  DERIVES  list  specified  for  procedure  advance 

[25]  [*  Advance  the  actual  input  stream  *] 

[26] 

[27]  PROCSPEC  consxme 

[28]  IMPLICIT  ( [**  IL  Global  Parameter  Section  **  ] 

[29]  INOUT  conslresult,  cons2result  :  integer 

[30]  INOUT  input  :  text); 

***  WARNING  :  no  DERIVES  list  specified  for  procedure  consume 


[*  Stream  operations  *] 

PROCSPEC  advance ( INOUT  eos  :  boolean, 

INOUT  next  ;  integer, 

IN  more  :  boolean) 

IMPLICIT  ( [**  IL  Global  Pareuneter  Section  **  ] 
INOUT  input  :  text ) ; 

no  DERIVES  list  specified  for  procedure  advance 
[*  Advance  the  actual  input  stream  *] 

PROCSPEC  consxune 

IMPLICIT  ( [*♦  IL  Global  Parameter  Section  **  ] 

INOUT  conslresult,  cons2result  :  integer 
INOUT  input  :  text); 

no  DERIVES  list  specified  for  procedure  consume 


[31] 

[•  Consume  the  input  stream  as 

[32] 

[•  (count  stream  elements  and 

[33] 

[35] 

MAINSPEC  (INOUT  input  :  text 

[36] 

INOUT  output  :  text ) ; 

[37] 

[38] 

PROC  advance; 

[40] 

VAR  ch :  char ; 

[42] 

#1 

IF  more  THEN 

[43] 

#3 

IF  eof _ text (input)  THEN 

[44] 

#5 

eos  : “  true 

[45] 

ELSE 

[46] 

#6 

eos  :“  false; 

[47] 

#7 

IF  eoln _ text (input)  THEN 

[48] 

#9 

text _ readln( input) ; 

[49] 

#10 

next  :-  cr 

Figure  B-2.  MALPAS  Intermediate  Language  Translation  of  Sample 
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[50] 

ELSE 

[51] 

#11 

text _ read _ char ( input ,  ch ) ; 

[52] 

#12 

next  ;=  char_pos(ch) 

[53] 

#8 

ENDIF 

[54] 

#4 

ENDIF 

[55] 

#2 

ENDIF 

[56] 

#ST0P 

[SKIP] 

[56] 

#END 

ENDPROC  [‘advance*] 

[57] 

[58] 

PRCC  consume; 

[60] 

VAR  eos _ 6:  boolean; 

[61] 

VAR  next _ 6:  integer; 

[63] 

#1 

conslresult  :=  0; 

[64] 

#2 

cons2result  :=  0; 

[65] 

#3 

[65] 

advance (eos _ 6,  next _ 6,  true); 

*** 

WARNING  : 

advance  has  not  been  fully  specified 

[66] 

#4 

LOOP  [while  loop] 

[67] 

#6 

EXIT  [while  loop]  WHEN  NOT(  NOT  eos _ 6); 

[68] 

#7 

conslresult  :=  conslresult  +  1; 

[69] 

[*  count  streeun  elements  *] 

[70] 

#8 

cons2result  :=  cons2result  +  next _ 6; 

[71] 

[*  Siam  stream  elements  *] 

[72] 

#9 

[72] 

advance (eos _ 6,  next _ 6,  true) 

*** 

WARNING  : 

advance  has  not  been  fully  specified 

[73] 

#5 

ENDLOOP  [while  loop] 

[74] 

#ST0P 

[SKIP] 

[74] 

#END 

ENDPROC  [‘consume*] 

[75] 

[76] 

MAIN 

[79] 

VAR  conslresult;  integer; 

[80] 

[*  result  returned  by  consumer  #1  *] 

[81] 

VAR  cons2result;  integer; 

[83] 

#1 

[83] 

consume ( ) ; 

*** 

WARNING  ; 

consume  has  not  been  fully  specified 

[84] 

#2 

text _ ^wr i t e  ( output ,  lit _ 1 _ theaverage )  ; 

[90] 

#ST0P 

[SKIP] 

[90] 

#END 

ENDMAIN 

[93] 

[““  WARNING  :  WARNINGS  IN  PASS  1  ...  See  Listing  File  ] 

[95] 

FINISH 

**•  WARNING  :  Procedure  body  for  text get  has  not  been  defined 

***  WARNING  :  Procedure  body  for  text page  has  not  been  defined 


***  WARNING  :  Procedure  body  for  text _ writeln  has  not  been  defined 

Figure  B-2.  MALPAS  Intermediate  Language  Translation  of  Sample(continued) 
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After  ONE-ONE,  13  nodes  removed. 

No  nodes  with  self-loops  removed. 

Node  id  No  of  pred.  Succ.  nodes 
# START  0  #END 

#END  1 

After  KASAI  (from  ONE-ONE) ,  No  nodes  removed. 

After  HECHT  ( from  ONE-ONE ) ,  No  nodes  removed . 

After  HK  (from  HE(^T) ,  No  nodes  removed. 

After  TOTAL  ( from  HR ) ,  No  nodes  removed . 

Control  Flow  Summary 

The  procedure  is  well  structured. 

The  procedure  has  no  unreachable  code  and  no  dynamic  halts . 

The  graph  was  fully  reduced  after  the  following  stages : 

ONE-ONE,  KASAI,  HECHT,  HK,  TOTAL 
The  graph  was  not  fully  reduced  after  the  following  stages : 

None 

Figure  B-3.  MALPAS  Control  Flow  Analysis  of  ADVANCE 

Key 

H  =  Data  read  and  not  subsequently  written  on  some  path  between  the  nodes 
I  =  Data  read  and  not  previously  written  on  some  path  between  the  nodes 
A  =  Data  written  twice  with  no  intervening  read  on  some  path  between  the  nodes 
U  =  Data  written  and  not  subsequently  read  on  some  path  between  the  nodes 
V  =  Data  written  and  not  previously  read  on  some  path  between  the  nodes 
R  *  Data  read  on  all  paths  between  the  nodes 
W  =  Data  written  on  all  paths  between  the  nodes 
E  =  Data  read  on  some  path  between  the  nodes 
L  Data  written  on  some  path  between  the  nodes 


After  ONE 

-ONE 

From 

To 

Data  Use 

Expression 

node 

node 

# START 

#END 

H 

ch 

input  more 

I 

input  more 

U 

eos 

input  next 

V 

ch 

eos  next 

R 

more 

E 

ch 

input  more 

L 

ch 

eos  input 

Summary  of  Possible  Errors 


No  errors  detected 


Figure  8-4.  MALPAS  Data  Use  Analysis  of  ADVANCE 


Information  Flow 


After  ONE-ONE 

From  node  # START  to  node  #END 


Identifier 


may  depend  on  identifier(s) 


eos 

INs/INOUTs 

eos 

input 

CONSTANTS 

false 

true 

next 

INs/INOUTs 

input 

more 

CONSTANTS 

cr 

input 

INs/INOUTs 

input 

more 

ch 

INs/INOUTs 

input 

more 

VARs/OUTs 

ch 

more 

next 


Identifier 


may  depend  on  conditional  node(s) 


eos 

#3 

#1 

next 

#7 

#3 

#1 

input 

#7 

#3 

#1 

ch 

#7 

#3 

#1 

Summary  of  Possible  Errors 
No  errors  detected 


Figure  B-S.  MALPAS  Information  Flow  Analysis  of  ADVANCE 


Semantic  Analysis 


After  ONE-ONE 

>From  node  ;  # START 
To  node  #END 

IF  NOT (more) 

THEN  MAP 
ENDMAP 

[ - ] 

ELSIF  more  AND  eof _ text (input) 

THEN  MAP 

eos  :=  true; 

ENDMAP 

[ - ] 

ELSIF  more  AND  eoln _ text(input)  AND  NOT(eof _ text(input)) 

THEN  MAP 

eos  :=  false; 
next  13; 

input  : =  readln text ( input ) ; 

ENDMAP 

[ - ] 

ELSIF  more  AND  NOT (eoln _ text (input))  AND  NOT (eof _ text (input)) 

THEN  MAP 

eos  :=  false; 

next  : “  char_pos ( read text char ( input ) ) ; 

input  :=  skip _ text _ char (input ); 

ch  !  "■  read _ text _ char  ( input )  ; 

ENDMAP  ENDIF 


Figure  B-6.  MALPAS  Semantic  Analysis  of  ADVANCE 
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APPENDIX  C 


SAMPLE  OUTPUTS  FROM  SoftTest 


/*  Test  while-loops  corresponding  to  the  code  template:  */ 


/*  iterate:  */ 

/*  while  not  done  do  */ 

/*  next_iteration;  */ 

/*  Test  using  the  following  expanded  logic:  */ 
/*  if  done_0  then  exit_0  */ 

/*  else  */ 

/*  do_f irst_iteration;  */ 

/*  if  done_l  then  exit_l  */ 

/*  else  •/ 

/*  do_second_iteration;  */ 

/*  if  done_2  then  exit_2  */ 

/*  else  ...  */ 


NODES 

Start 

DoneO 

ExitO 

DoFirst 

Donel 

Exitl 

DoSecond 

Done2 

Exit2 

Exit 


'Start  iteration  process'. 

'Iteration  process  has  completed  after  zero  iterations' 
'Exit  after  zero  iterations'. 

'Perform  first  step  of  iteration' . 

'Iteration  process  has  completed  after  first  iteration' 
'Exit  after  first  iteration'. 

'Perform  second  step  of  iteration' . 

'Iteration  process  has  completed  after  second  iteration 
'Exit  after  second  iteration'. 

'Exit  from  loop'  OBS. 


CONSTRAINTS 

excl( ExitO,  Exitl,  Exit2} . 
mask (not  Start,  DoneO). 
mask(not  DoFirst,  Donel). 
mask(not  DoSecond,  Done2) . 


RELATIONS 

ExitO 

DoFirst 

Exitl 

DoSecond 

Exit2 

Exit 


Start  and  DoneO . 

Start  and  not  DoneO . 
DoFirst  and  Donel . 
DoFirst  and  not  Donel. 
DoSecond  and  Done2 . 

ExitO  or  Exitl  or  Exit2. 


Figure  C-1.  SoftTest  Cause-Effect  Graph  Input 
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Functional  Variations  for:  Test  template  for  iteration  schemes 


1. 

If 

2. 

If 

3. 

If 

4. 

If 

5. 

If 

6. 

If 

7. 

If 

8. 

If 

9. 

If 

10. 

If 

11. 

If 

12. 

If 

13. 

If 

14. 

If 

15. 

If 

16. 

If 

17. 

If 

18. 

If 

19. 

If 

Start  and  DoneO  then  ExitO . 
not  Start  then  not  ExitO . 
not  DoneO  then  not  ExitO . 

Start  and  not  DoneO  then  DoFirst. 
not  Start  then  not  DoFirst. 

DoneO  then  not  DoFirst. 

DoFirst  and  Donel  then  Exitl. 
not  DoFirst  then  not  Exitl. 
not  Donel  then  not  Exitl. 

DoFirst  and  not  Donel  then  DoSecond. 
not  DoFirst  then  not  DoSecond. 

Donel  then  not  DoSecond. 

DoSecond  and  Done2  then  Exit2 . 
not  DoSecond  then  not  Exit2 . 
not  Done2  then  not  Exit2 . 

ExitO  then  Exit. 

Exitl  then  Exit. 

Exit2  then  Exit. 

not  ExitO  and  not  Exitl  and  not  Exit2  then  not  Exit. 


Complexity  Factors: 

variations  /  primary  causes  =  4.8 

variations  /  (primary  causes  +  primary  effects)  =  3.8 


Figure  C-2.  SoftTest  Variation  Analysis  Output 


Test  Cases/Coverage  Matrix  for:  Test  template  for  iteration  schemes 
TEST  CASE  1 : 

Cause  states : 

Start  •  Start  iteration  process 

DoneO  =  Iteration  process  has  completed  after  zero  iterations 

Effect  states: 

Exit  “  Exit  from  loop 


TEST  CASE  2 : 

Ceruse  states : 

Start  =  Start  iteration  process 

not  DoneO  =  not  Iteration  process  has  completed  after  zero  iterations 
Donel  =  Iteration  process  has  completed  after  first  iteration 

Effect  states: 

Exit  =  Exit  from  loop 


VARIATION  COVERAGE  by  test: 

Test  Variations 


1  1  16 

2  4  7  17 

3  4  10  13  18 

4  2  19 

5  3  9  15  19 

TEST  COVERAGE  by  variation: 
Var  Tests 


1  1 

2  4 

3  5 

4  2  3 

5  Untestable 

6  Untestable 

7  2 

19  4  5 

Tested  variations  /  testable  variations  -  100  % 

Figure  €-3.  SoftTest  Test  Synthesis  Output 
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Figure  C-4.  SoftTest  Cause-Effect  Graph 
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APPENDIX  D 


SAMPLE  OUTPUTS  FROM  TCAT-PATH,  TCAT/Ada,  and  S-TCAT/Ada 


—  TCAT-PATH/Ada,  Release  1.2  for  UNIX/SUN(tm)  (05/05/89) 

—  (c)  Copyright  1989  by  Software  Research,  Inc.  ALL  RIGHTS  RESERVED. 

—  SEGMENT  REFERENCE  LISTING  Fri  Sep  6  12:56:37  1991 

procedure  STORE_PATTERN  (  PAT_ID,  PAT:  in  LLATTRIBUTE  )  is 
—  Store  a  pattern  definition  in  the  pattern  table. 

—  Patterns  are  stored  in  alphabetical  order  by  name, 
begin 

—  START  instrumented  module  "STORE_PATTERN"  with  10  segment(s). 

—  >  ST0RE_PATTERN/1 : 

if  CUR_TABLE_ENTRIES  =  PATTERN_TABLE_SIZE  then 

—  >  ST0RE_PATTERN/2 : 

—  >  ST0RE_PATTERN/3 : 

raise  PATTERN_TABLE_FULL; 
end  if; 

for  I  in  1  . .  CUR_TABLE_ENTRIES  loop 

—  >  STORE_PATTERN/4 : 

CUR_TABLE_ENTRIES  :=  CUR_TABLE_ENTRIES  +  1; 
PATTERN_TABLE(CUR_TABLE_ENTRIES)  :=  PAT; 

PATTERN_TABLE ( CUR_TABLE_ENTRIES ) . NAME  : =  PAT_ID . STRING_VAL ; 
end  STORE_PATTERN; 

—  FINISH  instrumented  module  "STORE_PATTERN" . 

—  END  OF  TCAT/ADA  REFERENCE  LISTING 


—  TCAT/Ada,  Release  1.2  for  UNIX/SUN(tm)  (05/05/89) 

—  INSTRUMENTATION  STATISTICS 


Module 

#  segments 

#  statements  # 

Conditional 

MERGE_RANGES 

3 

0 

1 

ALTERNATE 

17 

0 

6 

CHAR_RANGE 

5 

0 

2 

RESTRICT 

22 

4 

825559569 

TAIL 

17 

4 

6 

RESOLVE_AMBIGUITY  20 

8 

288143 

COMPLETE_ALT 

14 

5 

288510 

REPEAT 

4 

0 

839518514 

STORE_PATTERN 

10 

0 

839518517 

ALTERNATE 

1 

0 

302080 

Figure  D-1.  TCAT-PATH  Reference  Listing 
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—  TCAT-PATH/Ada,  Release  1.2  for  UNIX/SUN(tm)  (05/05/89) 

—  (c)  Copyright  1989  by  Software  Research,  Inc.  ALL  RIGHTS  RESERVED. 

—  Instrumented  module  names  Fri  Sep  6  12:56:37  1991 

MERGE_RANGES  3 

CHAR_RANGE  5 
RESTRICT  22 
TAIL  17 

RESOLVE_AMBIGUITy  20 
COMPLETE_ALT  14 

COMPLETE_CONCAT  13 
C0MPLETE_0PT  4 

COMPLETE_PAT  17 

COMPLETE_PATTERNS  3 
CONCATENATE  4 
CVT_ASCII  11 
CVT_STRING  7 
EMIT_ADVANCE_HDR  1 
EMIT_ADVANCE_TLR  1 
EMIT_PKG_DECLS  5 
EMIT_ALT_CASES  7 
EMIT_CONCAT_CASES  8 
EMIT_C0NCAT_R1GHT  5 
EMIT_PATTERN_MATCH  32 

EMIT_CHAR  12 
EMIT_SELECT  17 
EM1T_SCAN_PR0C  7 
EMIT_TOKEN  18 
INCLUDE_PATTERN  3 
LOOK_AHEAD  1 
LOOK_UP_PATTERN  5 
OPTION  4 

REPEAT  4 

STORE_PATTERN  10 

Figure  D-2.  TCAT-PATH  Instrumentation  Counts  for  STORE_PATrERN 


—  TCAT-PATH/Ada,  Release  1.2  for  UNIX/SUN(tm)  (05/05/89) 

—  (c)  Copyright  1989  by  Software  Research,  Inc.  ALL  RIGHTS  RESERVED. 

—  ERROR  LISTING  Fri  Sep  6  12:56:37  1991 

— Module  EMIT_PATTERN_NAME  overloaded,  not  instrumented 
Internal  error  loop_edge_back  called  with  no  edges  to  loop  back 
— Module  ALTERNATE  overloaded,  not  instrumented 


Figure  D-S.  TCAT-PATH  Error  Listing  for  STORE_PATTERN 
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Figure  D-4.  TCAT-PATH  Digraph  of  STORE_PATTERN 
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Path  Analysis  Statistics 

File  name:  STORE_PATTERN.dig 

Number  of  nodes :  6 

Number  of  edges:  10 

Cyclomatlc  number  ( E  -  N  +  2 ) :  6 

Number  of  paths :  10 

Average  path  length  ( segments ) : 

Minimum  length  path  (segments); 

Maximum  length  path  (segments); 

Most  iteration  groups : 

Path  count  by  iteration  groups: 

0  iteration  group ( s ) :  2 

1  iteration  group ( s ) :  8 

Stopped  at  1  iteration  groups 


Figure  D-5.  TCAT-PATH  Path  Analysis  of  STORE-PATTERN 
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cyclo  [Release  3.3  —  9/26/90] 

Cyclomatic  Number  =  Edges  -  Nodes  +2  =  10-6+2  =  6 

Figure  D-6.  TCAT-PATH  Complexity  of  STORE_PATTERN 


Ct  Test  Coverage  Analyser  Version  1 . 8 

(c)  Copyright  1990  by  Software  Research,  Inc. 

Module  "STORE_PATTERN" :  10  paths,  2  were  hit  in  5  invocations. 
20.0%  Ct  coverage 
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Figure  D-7.  TCAT-PATH  Coverage  Report  for  STORE_PATTERN 


—  TCAT-PATH/Ada ,  Release  1.2  for  UNIX/SUN(tm)  (05/05/89) 

—  (c)  Copyright  1989  by  Software  Research,  Inc.  ALL  RIGHTS  RESERVED. 

—  SEGMENT  REFERENCE  LISTING  Fri  Sep  6  12:56:37  1991 

procedure  STORE_PATTERN  (  PAT_ID,  PAT:  in  LLATTRIBUTE  )  is 
Store  a  pattern  definition  in  the  pattern  table. 
Patterns  are  stored  in  alphabetical  order  by  ncime. 

begin 

—  START  instrumented  module  "STORE_PATTERN"  with  10  segment(s) . 

—  >  STORE_PATTERN/l : 

if  CUR_TABLE_ENTRIES  =  PATTERN_TABLE_SIZE  then 

—  >  ST0RE_PATTERN/2 : 

—  >  ST0RE_PATTERN/3 : 

raise  PATTERN_TABLE_FULL; 
end  if; 

for  I  in  1  . .  CUR_TABLE_ENTRIES  loop 

—  >  ST0RE_PATTERN/4 : 

CUR_TABLE_ENTRIES  :=  CUR_TABLE_ENTRIES  +  1; 
PATTERN_TABLE(CUR_TABLE_ENTRIES)  :=  PAT; 

PATTERN_TABLE ( CUR_TABLE_ENTRIES  >  . NAME  : =  PAT_ID . STRING_VAL ; 
end  STORE_PATTERN; 

—  FINISH  instrumented  module  "STORE_PATTERN" . 

—  END  OF  TCAT/ADA  REFERENCE  LISTING 


—  TCAT/Ada,  Release  1.2  for  UNIX/SUN(tm)  (05/05/89) 

—  INSTRUMENTATION  STATISTICS 


—  Module 


#  segments 


#  statements  #  Conditional  statements 


—  MERGE_RANGES 

3 

0 

1 

—  ALTERNATE 

17 

0 

6 

—  CHAR_RANGE 

5 

0 

2 

—  RESTRICT 

22 

4 

825559569 

—  TAIL 

17 

4 

6 

—  RESOLVE_AMBIGUITY 

20 

8 

288143 

—  COMPLETE_ALT 

14 

5 

288510 

—  COMPLETE_CONCAT 

13 

6 

839518519 

—  REPEAT 

4 

0 

839518514 

—  STORE_PATTERN 

10 

0 

839518517 

—  ALTERNATE 

1 

0 

302080 

Figure  D-8.  TCAT/Ada  Reference  Listing 
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Figure  D-9.  TCAT/Ada  Coverage  Report 
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Figure  D-9.  TCAT/Ada  Coverage  Report  (continued) 
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Cl  Segment  Hit  Report. 
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Segment  Level  Histogram  for  Module:  BUILDRIGHT 


+ - + 

I  Number  of  Executions,  Normalized  to  Maximum 
I  (Maximum  =  348  Hits)  X  =  One  Hit 

I  (Scale:  0.287  Each  X  =  6.960  Hits) 

Segment  Number  Of  | 

Number  Executions  >-l - 20 - 40 - 60 - 80 - 100 

+ - + - + 


1 

2 

3 

4 

5 

6 

7 

8 

9  * 

10 
11 
12 

13 

14  * 

15 


128 

348 

348 

100 

122 

92 

26 

8 


XXXXXXXXXXXXXXXXXX 

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 

xxxxxxxxxxxxxx 

xxxxxxxxxxxxxxxxx 

xxxxxxxxxxxxx 

XXX 

X 


158 

190 

288 

60 


xxxxxxxxxxxxxxxxxxxxxx 

xxxxxxxxxxxxxxxxxxxxxxxxxxx 

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 

xxxxxxxx 


128 


XXXXXXXXXXXXXXXXXX 


+ 


(* 


- + 

Zero  Hits) 


+ 


Average  Hits  per  Executed  Segment:  153.5385 
Cl  Value  for  this  Module:  86.6667 


Figure  D-9.  TCAT/Ada  Coverage  Report  (continued) 


—  S-TCAT/Ada,  Release  1.2  for  UNIX/SUN(tm)  (05/05/89) 

—  (c)  Copyright  1989  by  Software  Research,  Inc.  ALL  RIGHTS  RESERVED. 

—  SEGMENT  REFERENCE  LISTING  Mon  Sep  9  13:51:16  1991 

with  LL_DECLARATIONS,  TEXT_I0,  INTEGER_TEXT_IO; 
package  body  LL_SUPPORT  is 

procedure  STORE_PATTERN  (  PAT_1D,  PAT:  in  LLATTRIBUTE  )  is 
—  Store  a  pattern  definition  in  the  pattern  table. 

—  Patterns  are  stored  in  alphabetical  order  by  name, 
begin 

—  START  instrumented  module  "STORE_PATTERN"  with  1  call  pairs(s). 

if  CUR_TABLE_ENTRIES  “  PATTERN_TABLE_SIZE  then 
—  1  guess  I  didn't  make  the  table  big  enough, 

raise  PATTERN_TABLE_FULL; 

PATTERN_TABLE(I)  :=  ALTERNATE (  PAT,  PATTERN_TABLE ( I )  ); 

—  >  ST0RE_PATTERN/1 : 

PATTERN_TABLE { I ) . NAME  : =  PAT_1D . STR1NG_VAL ; 
return / 
end  if; 
end  loop; 

CUR_TABLE_ENTRIES  :=  CUR_TABLE_ENTRIES  +  1; 
PATTERN_TABLE(CUR_TABLE_ENTRIES)  :=  PAT; 

PATTERN_TABLE ( CUR_TABLE_ENTRIES ) . NAME  : =  PAT_I D . STRING_VAL ; 
end  STORE_PATTERN; 

—  FINISH  instrumented  module  "STORE_PATTERN" . 

—  END  OF  S-TCAT/ADA  REFERENCE  LISTING 


—  S-TCAT/Ada,  Release  1.2  for  UNIX/SUN(tm)  (05/05/89) 


—  INSTRUMENTATION  STATISTICS 


—  Module 


»  segments 


#  statements 


MERGE_RANGES 

ALTERNATE 

CHAR_RANGE 

RESTRICT 

TAIL 

RESOLVE_AMBIGUITY 

COMPLETE_ALT 

STORE_PATTERN 
LL  SUPPORT 


0  0 

5  0 

0  0 

9  4 

14  4 

28  8 

3  5 

1  0 

0  0 


Figure  D-10.  S-TCAT/Ada  Reference  Listing 
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—  S-TCAT/Ada,  Release  1.2  for  UNIX/SUN(tm)  (05/05/89) 

—  (c)  Cppyright  1989  by  Software  Research,  Inc.  ALL  RIGHTS  RESERVED. 

—  Instrumented  module  names  Mon  Sep  9  13:51:16  1991 

MERGE_RANGES  0 

ALTERNATE  5 
CHAR_RANGE  0 
RESTRICT  9 
TAIL  14 

RESOLVE_AMBIGUITY  28 
COMPLETE_ALT  3 

COMPLETE_PATTERNS  0 
CONCATENATE  0 

REPEAT  0 

STORE_PATTERN  1 
LL_SUPPORT  0 
MERGE_RANGES  0 

ALTERNATE  5 
CHAR_RANGE  0 
RESTRICT  9 
TAIL  14 

RESOLVE_AMBIGUITY  28 
COMPLETE_ALT  3 

COMPLETE_PATTERNS  0 
CONCATENATE  0 
CVT_ASCII  0 

cvt'string  1 

EMIT_ADVANCE_HDR  0 
EMIT_ADVANCE_TLR  0 
EMIT_PKG_DECLS  0 
EMIT_ALT_CASES  4 
EMIT_CONCAT_CASES  3 
EMIT_CONCAT_RIGHT  2 
EMIT_PATTERN_MATCH  13 

EMIT_CHAR  0 
EMIT_SCAN_PROC  3 
EMIT_TOKEN  0 
INCLUDE_PATTERN  1 
OPTION  0 

REPEAT  0 

STORE_PATTERN  1 
LL  SUPPORT  0 


Figure  D*ll.  S-TCAT/Ada  Instrumentation  Counts 
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alternate  alternate 
alternate  alternate 

ALTERNATE  ALTERNATE 
ALTERNATE  MERGE_RANGES 
ALTERNATE  MERGE_RANGES 
RESTRICT  RESTRICT 

RESTRICT  RESTRICT 

RESTRICT  ALTERNATE 
RESTRICT  RESTRICT 
RESTRICT  CONCATENATE 
RESTRICT  RESTRICT 
RESTRICT  OPTION 
RESTRICT  OPTION 
RESTRICT  CONCATENATE 
TAIL  ALTERNATE 
TAIL  TAIL 
TAIL  TAIL 
TAIL  CONCATENATE 
TAIL  CONCATENATE 
TAIL  ALTERNATE 
TAIL  TAIL 
TAIL  TAIL 
TAIL  CONCATENATE 
TAIL  TAIL 
TAIL  CONCATENATE 
TAIL  TAIL 
TAIL  ALTERNATE 
TAIL  TAIL 

RESOLVE_AMBIGUITY  RESTRICT 
RESOLVE_AMBIGUITY  RESTRICT 
RESOLVE_AMBIGUITY  ALTERNATE 
RESOL VE_AMBIGUITY  ALTERNATE 
RESOLVE_AMBIGUITY  ALTERNATE 
RESOLVE_AMBIGUITY  ALTERNATE 
RES0LVE_AMBIGUITY  TAIL 
RESOL VE_AMBIGUITY  TAIL 
RESOLVE_AMBIGUITY  OPTION 
RESOLVE_AMBIGUITY  CONCATENATE 
RESOLVE_AMBIGUITY  OPTION 
RESOLVE_AMBIGUITY  CONCATENATE 
RESOLVE_AMBIGUITY  CONCATENATE 
RESOLVE_AMBIGUITY  ALTERNATE 
RESOL VE_AMBIGUITY  CONCATENATE 
RESOLVE_AMBIGUITY  ALTERNATE 
RESOLVE_AMBIGUITY  CONCATENATE 

STORE  PATTERN  ALTERNATE 


Figure  D-t2.  S-TCAT/Ada  Instrumented  Call-Pairs 
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—  S-TCAT/Ada,  Release  1.2  for  UNIX/SUN(tm)  (05/05/89) 

—  (c)  Copyright  1989  by  Software  Research,  Inc.  ALL  RIGHTS  RESERVED. 
--  ERROR  LISTING  Mon  Sep  9  13:51:16  1991 

EMIT_PATTERN_NAME  not  instrumented  because  overloaded 
COMPLETE_PAT  not  instrumented  because  overloaded 
COMPLETE_CONCAT  not  instrumented  because  overloaded 
COMPLETE_OPT  not  instrumented  because  overloaded 
EMIT_SELECT  not  instrumented  because  overloaded 
L00K_UP_PATTERN  not  instrumented  because  overloaded 
— Module  EM1T_PATTERN_NAME  overloaded,  not  instrumented 
L00K_AHEAD  not  instrumented  because  overloaded 

Figure  D-13.  S-TCAT/Ada  Error  Listing 


eg  1.12 

Root  node  is  ALTERNATE 
(  1 )  ALTERNATE 

(2)  ALTERNATE  (*  recursive  cycle  *) 

(  2 )  ALTERNATE  ( *  recursive  cycle  • ) 

(2)  .  ALTERNATE  (*  recursive  cycle  *) 

(2)  .  MERGE_RANGES 

(2)  .  MERGE_RANGES 

Graph  contains  3  cycles. 


Disconnected  nodes : 

RESTRICT 

CONCATENATE 

OPTION 

TAIL 

RESOLVE_AMBIGUITY 

COMPLETE_ALT 

CVT_STRING 

EMIT_ALT_CASES 

EMIT_PATTERN_MATCH 

EMIT_CONCAT_CASES 

EMIT_CONCAT_RIGHT 

EMIT_SCAN_PROC 

INCLUDE_PATTERN 

STORE  PATTERN 


Figure  D>14.  S-TCAT/Ada  Control  Graph 
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Coverage  Analyzer.  [Release  7.3  for  SUN/UNIX  12/90] 
(c)  Copyright  1990  by  Software  Research,  Inc. 

Selected  SCOVER  System  Option  Settings: 


[-C] 

Cumulative  Report 

—  NO 

[-p] 

Past  History  Report 

—  YES 

[-n] 

Not  Hit  Report 

—  YES 

[-H] 

Hit  Report 

—  NO 

[-nh] 

Newly  Hit  Report 
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Figure  D*15.  S-TCAT/Ada  Coverage  Report 
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S-TCAT:  Coverage  Analyzer.  [Release  7.3  for  SUN/UNIX  12/90] 
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SI  Not  Hit  Report. 


1 

EMIT_TOKEN 

All 

Call-pairs 

Hit. 

SI  =  100% 

2 

CHAR_RANGE 

All 

Call-pairs 

Hit. 

SI  =  100% 

3 

CONCATENATE 

All 

Call-pairs 

Hit. 

SI  =  100% 

4 

ALTERNATE 

1 

2  .3 

4 

5 

5 

STORE_PATTERN 

1 

6 

OPTION 

All 

Call-pairs 

Hit. 

SI  =  100% 

7 

COMPLETE_PATTERNS 

All 

Call-pairs 

Hit. 

SI  =  100% 

8 

EMIT_PKG_DECLS 

All 

Call-pairs 

Hit. 

SI  =  100% 

9 

EMIT_ADVANCE_HDR 

All 

Call-pairs 

Hit. 

SI  =  100% 

10 

INCLUDE_PATTERN 

All 

Call-pairs 

Hit. 

SI  =  100% 

11 

EMIT_ADVANCE_TLR 

All 

Call-pairs 

Hit. 

SI  =  100% 

12 

EMIT_SCAN_PROC 

2 

3 

13 

EMIT_CHAR 

All 

Call-pairs 

Hit. 

SI  =  100% 

14 

EMIT_PATTERN_MATCH 

2 

3  4 

5 

6  7  8 

9 

11 

12  13 

15 

EMIT_CONCAT_RIGHT 

2 

16 

EMIT_CONCAT_CASES 

2 

3 

17 

CVT_STRING 

All 

Call-pairs 

Hit. 

SI  =  100% 

18 

REPEAT 

All 

Call-pairs 

Hit. 

SI  =  100% 

19 

COMPLETE_ALT 

2 

3 

20 

RESOLVE_AMBIGUITy 

2 

3  4 

5 

6  7  8 

9 

11 

12  13 

14 

15  16  17 

18 

20 

21  22 

23 

24  25  26 

27 

21 

RESTRICT 

2 

3  4 

5 

6  7  8 

9 

22 

TAIL 

1 

2  3 

4 

5  6  7 

8 

10 

11  12 

13 

14 

23 

EMIT_ALT_CASES 

2 

3  4 

Number 

of  Call-pairs  Not  Hit 

: 

77 

Total  Number  of  Call-pairs : 

87 

SI  Coverage  Value: 

11.49% 

S-TCAT 

:  Coverage  Analyzer . 

[Release  7.3  for 

SUN/UNIX  12/90] 

(c)  Copyright  1990  by  Software  Research,  Inc. 

Call-pair  Level  Histogram  for  Module:  ALTERNATE 

No  segments  hit 

S-TCAT:  Coverage  Analyzer.  [Release  7.3  for  SUN/UNIX  12/90] 
(c)  Copyright  1990  by  Software  Research,  Inc. 

Call-pair  Level  Histogram  for  Module:  STORE_PATTERN 

No  segments  hit 


10 


10 

19 

28 


Figure  D-15.  S-TCAT/Ada  Coverage  Report  (continued) 
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S-TCAT:  Coverage  Analyzer.  [Release  7.3  for  SUN/UNIX  12/90] 
S-TCAT:  Coverage  Analyzer.  [Release  7.3  for  SUN/UNIX  12/90] 
(c)  Copyright  1990  by  Software  Research,  Inc. 

Call-pair  Level  Histogrenn  for  Module:  INCLUDE_PATTERN 


+• 


+ 


Call-pair  Number  Of  1 
Number  Executions  > 


20 


Number  of  Executions,  Normalized  to  Maximum 
(Maximum  =  20  Hits)  X  =  One  Hit 

(Scale:  5.000  Each  X  =  0.400  Hits) 

■1 - 20 - 40 - 60 - 80 - 100 


xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 


Average  Hits  per  Executed  Call-pair:  20.0000 
SI  Value  for  this  Module:  100.0000 


S-TCAT:  Coverage  Analyzer.  [Release  7.3  for  SUN/UNIX  12/90] 
(c)  Copyright  1990  by  Software  Research,  Inc. 

Call-pair  Level  Histogram  for  Module:  EMIT_ALT_CASES 


+ - + 

I  Number  of  Executions,  Normalized  to  Maximum 
I  (Maximum  =  12  Hits)  X  =  One  Hit 

I  (Scale:  8.333  Each  X  =  0.240  Hits) 

Call-pair  Number  Of  | 

Number  Executions  >-l - 20 - 40 - 60 - 80 - 100 


1  12  I  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 

2  *  I 

3  *  I 


(*  =  Zero  Hits) 

Average  Hits  per  Executed  Call-pair:  12.0000 
SI  Value  for  this  Module:  25.0000 


Figure  D-15.  S-TCAT/Ada  Coverage  Report  (continued) 
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{%c  Values  file  1:  for  variable  number  of  initial  TDGen  executions.  ) 


expr 

op 

identifier 

real_no 

integl 

integ2 


{%  expr)£%  op){%  expr)  {%  identifier)  {%  real_no) 

+  -  /  * 

variablel  variable2  {%  alpha  6) 

[%  real  4.6)  [%  integl)E+{%  integ2)  [%  integl)E-l%  integ2) 
{%r  1. .100) 

{%r  3. .6) 


{%c  Values  file  2:  for  last  two  executions  of  TDGen.  ) 


expr 

op 

identifier 

real_no 

integl 

integ2 


variablel  variable2  {%  alpha  6)  [%  real  4.6)  [ 

+  -  /  * 

variablel  variable2  [%  alpha  6) 

[%  real  4.6)  {%  integl)E+[%  integ2)  (%  integl)E-[%  integ2) 
{%r  1. . 100) 

[%r  3. .6) 


{%c  Template  file:  Produces  arithmetic  expression  of  varying  ) 
{%c  complexity  for  use  in  testing  a  generated  lexical  analyzer.  ) 

{ %  expr  ) 


Figure  D-16.  TDGen  Sample  Value  and  Template  Files 


Field 

No.  Table 
Entries 

Cumulative  Total 

Combinations 

%  expr 

3 

3 

%  op 

4 

12 

%  identifier 

3 

36 

%  real_no 

3 

108 

%  integl 

100 

10800 

%  integ2 

4 

43200 

integl )E+[% 


Figure  D-17.  TDGen  Table  of  Sequential  Combinations  for  Initial  Files 


{%  real_no) 

{ %  expr ) { %  op ) { %  expr ) 
{%  identifier} 

{ %  expr } { %  op } { %  expr } 
{%  identifier) 

{%  identifier) 

{%  expr){%  op){%  expr) 
{%  real_no) 

{%  real_no} 

{%  identifier) 


Figure  D-18.  TDGen  Output  of  First  Random  Execution 


3E+6 

{%  real  4 . 6 )-variablel 
RSBEZ4 

{%  integl)E-[%  integ2)-[%  integl)E-{%  integ2) 

variable2 

variable2 

{%  identif ier) * (%  real_no)/[%  identif ier)/{%  identifier) 

21E+4 

47E-6 

variablel 


Figure  D-19.  TDGen  Output  After  3  Executions  with  1st  Value  File 


3E+6 

3092 . 703258-variablel 

RSBEZ4 

53E-4-83E-6 

variable2 

variable2 

G36dk5*26E-5/clinHEJ/variable2 

21E+4 

47E-6 

variablel 


Figure  D-20.  TDGen  Output  After  2  Executions  with  2nd  Value  File 
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APPENDIX  E 


SAMPLE  OUTPUTS  FROM  TBGEN  AND  TCMON 


—  Script  file  :  USR: [ADATEST. TBGEN] CALENDAR. REC;1 

—  Created  at  :  1991-08-15  10:37:14 

—  Created  by  :  Test  bed  CAL_BED  generated  at  1991-08-15  09:00:03 
SET  TRACE  FILE  calendar. trc 

DECLARE 

USE  calendar 

moment  :  time  : =  clock 

current_year  :  year 
current_month  :  month 
the_day  :  day_num 

seconds  :  day_dura 

BEGIN 

split (moment ,  current_year ,  current_month ,  the_day,  seconds) 


moment  :=  time_of (current_year,  current_month,  15,  0.0) 
DISPLAY  day (moment) 

morient  :=  add _ l(moment,  86400.0)  —  add _ 1  equiv  to  "  +  " 

split(moment,  current_year ,  current_month,  the_day,  seconds) 
ASSERT  the_day  =  16  AND  «:oconds  =  0.0 


now  :  time  : =  clock 

later  :  time  : =  clock 

ASSERT  le _ l(now,  later)  =  true  —  le _ 1  equiv  to  "<=” 

moment  :=  time_of (1991,  2,  28,  0.0) 

ASSERT  NOT  EXCEPTION 

moment  :=  time_of ( 1991,  2,  29,  0.0) 

ASSERT  EXCEPTION (time_error) 

END 

SET  TRACE  CLOSED 
SET  RECORD  CLOSED 


Figure  E-1.  TBGEN  Record  File 
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*1Hlt****<r*H********1Hk**»****^************1k**»*»*1k*1Hlr***********lk***<r**1»r*r***» 

*  Softplan  (R)  Ada  Tools  » 

*  TBGEN  System  Version  3.1,  Copyright  (C)  1990  Nokia  Data  Systems  * 

*  Test  Bed  Trace  Listing  * 

********«*******************«**********tlr********************************«** 

Test  bed  generated  at  1991-08-15  09:00:03.  Time  is  now  1991-08-15  10:37:22 
CAL_BED>  DECLARE 
CAL_BED>  USE  calendar 

CAL_BED>  moment  :  time  :=  clock 

CAL_BED>  current_year  :  year 

CAL_BED>  current_month  :  month 

CAL_BED>  the_day  :  day_num 

CAL_BED>  seconds  :  day_dura 

CAL_BED>  BEGIN 

CAL_BED>  split(moment,  current_year,  current_month,  the_day,  seconds) 

YEAR  (out)  =  1991 

MONTH  (out)  =  8 

DAY  (out)  =  15 

SECONDS  (out)  =  38266.8500 

CAL_BED> 

CAL_BED>  moment  :=  time  of (current  year,  current_month ,  15,  0.0) 

CAL_BED>  DISPLAY  day (moment) 

15 

CAL_BED>  moment  :=  add _ l(moment,  86400.0)  —  add _ 1  equiv  to  "+" 

CAL_BED>  split(moment,  current_year ,  current_month,  the_day,  seconds) 

YEAR  (out)  =  1991 

MONTH  (out)  =  8 

DAY  (out)  =  16 

SECONDS  (out)  =  0.0000 

CAL_BED>  ASSERT  the_day  =  16  AND  seconds  =0.0 
CAL_BED> 

CAL_BED>  now  :  time  :=  clock 

CAL_BED>  later  :  time  :=  clock 

CAL_BED>  ASSERT  le _ l(now,  later)  =  true  —  le _ 1  equiv  to  "<=" 

CAL_BED> 

CAL_BED>  moment  :=  time_of ( 1991,  2,  28,  0.0) 

CAL_BED>  ASSERT  NOT  EXCEPTION 
CAL_BED> 

CAL_BED>  moment  :=  time_of (1991,  2,  29,  0.0) 

*»*  exception  CALENDAR. TIME_ERROR 
CAL_BED>  ASSERT  EXCEPTION (time_error) 

CAL_BED>  END 

CAL_BED>  SET  TRACE  CLOSED 

Trace  closed  at  1991-08-15  10:43:46 


Figure  E-2.  TBGEN  Trace  File 
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*  ♦♦★★lUClfc*************************************!^***************************** 

*  Softplan  (R)  Ada  Tools  * 

*  TBGEN  System  Version  3.1,  Copyright  (C)  1990  Nokia  Data  Systems  * 

*  Test  Bed  Generation  Log  File  * 

Ik*****!********************************************************************* 

Licence  identification  of  the  generator: 

Test  bed  timesteuap. . . :  1991-08-15  09:00:03 


Test  bed  neune . :  CAL_BED 

Generated  Ada  files..:  cal*.ada 
Command  file . :  calCMD.COM 


Analysed  source  files : 
File:  TBGENSYS.STD 
File :  calendar . spe 


The  symbol  table  : 

package  STANDARD/8001/  is 
type  BOOLEAN/1/  is  ( 

FALSE, 

TRUE) ; 

type  INTEGER/2/  is  Integer_Type; 
type  FLOAT/3/N0V/  is  Float_Type; 
type  CHARACTER/4/N0V/  is  ( 

<>); 

subtype  NATURAL/5/NoV/  is  INTEGER  <2>  ; 

—  Type  Class  =>  Integer_Type 
subtype  POSITIVE/6/N0V/  is  INTEGER  <2>  ; 

—  Type  Class  =>  Integer_Type 

function  ">="/2015/( 

LEFT  :  in  CALENDAR. TIME  <11>  ; 

RIGHT  :  in  CALENDAR. TIME  <11>  ) 
return  BOOLEAN  <1>  ; 

—  Alias  Name  =>  GE _ 1 

TIME_ERROR/6006/  :  exception; 

end  CALENDAR; 
end  STANDARD; 

Execution  of  the  generator  successfully  ended  at  1991-08-15  09:01:02 


Figure  E-3.  TBGEN  Generated  Log  File 
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*************************************************************** 

*  TCMON  System  Version  2.2,  (C)  Copyright  1987  by  Softplan  • 

*  Test  Coverage  Monitor  /  Progreun  Bottleneck  Finder  * 

*  Execution  profile  listing  * 

************************************************************** 

Counters  Timers 


LINE  EXECUTION  LE-  PLACE 

NO.  COUNTS  VEL  DESCRIPTION 

Source  file  =>  . adalex2] ll_decls . ada 

Source  file  =>  ll_compile_dummy . ada 


START/  END/ 
TRUE  FALSE 


23  . 

120 . 

124*****>>>>> 

127*****>>>>>>> 


1  proc  LL_COMPILE 

2  func  LLFIND 

2  begin 

3  while_loop 

Condition 


198 

898 

898 


Source  file  =>  [- . adalex2] ll_sup_spec . ada 
Source  file  =>  ll_sup_body_mt . ada 

50 .  1  pack  LL_SUPPORT 

97 .  2  func  ALTERNATE 

170 .  2  func  CHAR_RANGE 

175*****>  2  begin  6 

176  T IMER_CHAR_RANGE 

178***  3  if_then  3 

Condition  3 

181***  3  if_else  3 

183*****>>>>  4  for_loop  62 

188*****>  3  return  6 

204 .  3  proc  COMPLETE_ALT 

Median  of  nonzero  counter  values  =  8 

One  asterisk  (  *  )  <=>  1 

Number  of  counters  =  539 

Number  of  timers  36 

Number  of  instrumented  (sub) conditions  ^  206 
Minimum  measurable  time  interval  - 

Estimated  cost  of  one  timer  operation  = 
Estimated  cost  of  one  counter  operation  = 

The  instrvimentation  was  done  1991-08-14  13:15; 
This  listing  was  produced  1991-08-14  13:32; 


0  ? 
763  ? 
63 


0 

14 

6 

0 

6 

3 

3 

3 

62 

0 

4 


27 

48 


AVERAGE  TOTAL 

TIME  TIME 

Instr  =>  (A,N,N,N) 

Instr  =>  (A,N,N,Y) 


Instr  =>  (A,N,N,N) 

Instr  =>  (A,Y,Y,y) 

0.0000 


0.0000 

0.0013 

0.0013 


0.0000 

0.0078 

0.0078 


0.0039 


0.0156 


0001 

0006 

0000 


Data  files: 

NAME  =>s2unpleTIM.  dat 


Figure  E-4.  TCMON  Profile  Execution  Listing 
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*4r*********W***tk4n*********«*******««;*«ik**lk*ik*lk**ilr************* 

*  TCMON  System  Version  2.2,  (C)  Copyright  1987  by  Softplan  » 

*  Test  Coverage  Monitor  /  Program  Bottleneck  Finder  * 

*  Log  of  TCMON  preprocessor  execution  * 

*******««**W**********************«****^******«*******lk******* 

Date  and  time  =>  1991-08-14  13:15:27 


Prefix  =>  SAMPLE 

Generated  files  =>  sample*. ada 

Main  procedure  =>  *not  specified* 

Code  pattern  file  =>  PATTERNS. TCM 

Source  =>  ll_sup_body_mt . ada 
Target  =>  s2Uivple_ll_sup_body_mt .  ada 
Instrumentation  =>  (  INC_M0DE  =>  UNCHECKED 

COUNTERS  =>  ALL 

AUTO_TIMERS  =>  YES 

MANUAL_TIMERS  =>  YES 

EXPAND_COMMANDS  =>  YES  ) 

package  body  LL_SUPPORT  on  line  50 
function  ALTERNATE  on  line  97 

function  MERGE_RANGES  on  line  103 
function  CHAR_RANGE  on  line  170 

— &&  start  timer_char_range  on  line  176  expanded 
— &&  stop  timer_char_range  on  line  187  expanded 
procedure  COMPLETE_PAT  on  line  192 


Summary  Information 

*«««•*********««;*«« 


Number  of  instrumented  files  =  6 
Number  of  compilation  units  =  4 
Number  of  body  stubs  =  2 
Number  of  subunits  =  2 
Number  of  statement  list  counters  =  539 
Number  of  (sub) condition  counters  206 
Number  of  timers  =  36 
Number  of  manual  timer  STARTs  =  2 
Number  of  manual  timer  STOPs  °  2 
Number  of  embedded  commands  =  1 


all  expanded 
all  expanded 
all  expanded 


Command  file  for  compilation  and  linking  =>  SAMPLECMD.COM 


ERRORS:  0  WARNINGS:  0 


Figure  El-5.  TCMON  Log  File 


83 


*  TCMON  System  Version  2.2,  (C)  Copyright  1987  by  Softplan  * 

*  Test  Coverage  Summary  Report  • 

************************************************************************ 


PROGRAM 

STM 

COND 

SUB 

OVER 

UNIT 

LIST 

CVRG 

COND 

ALL 

CVRG 

CVRG 

CVRG 

Source  file  =>  ll_compile_dummy . ada 

Instr 

=  >  (A, 

N,N,Y) 

proc  LL_COMPILE 

func  LLFIND 

88  - 

88  - 

88  - 

88 

- 

proc  LLPRTSTRING 

0  - 

0  - 

0  - 

0 

- 

proc  LLPRTTOKEN 

0  - 

0  - 

0  - 

0 

- 

proc  LLSKIPTOKEN 

0  - 

0 

- 

proc  LLSKIPNODE 

0  - 

0 

- 

proc  LLSKIPBOTH 

0  - 

0 

- 

proc  LLFATAL 

0  - 

0 

- 

proc  GET_CHARACTER 

0  - 

0  - 

0  - 

0 

- 

func  MAKE_TOKEN 

0  - 

0  - 

0  - 

0 

- 

proc  LLNEXTTOKEN 

100 

100 

100 

100 

proc  LLMAIN 

62  - 

47  - 

47  - 

56 

- 

body  LL_COMPILE 

100 

100 

proc  LL_COMPILE 

49  - 

44  - 

45  - 

48  - 

Source  file  =>  [- 

. adalex2] ll_tokens . ada 

Instr  => 

(A,N,N,N) 

pack  LL_T0KENS 
proc  ADVANCE 

83  - 

71  - 

74  - 

79  - 

pack  LL_T0KENS 

83  - 

71  - 

74  - 

79  - 

OVERALL  SUMMARY  : 

46  - 

44  - 

SSKSSSSSSS 

44  - 

46  - 

Number  of  partially  instrumented  or  dropped  compilation  units  :  0 


This  summary  was  generated  at  1991-08-14  13:34:48,  based  on  the  TCMON 
execution  profile  listing  file  sample_out.dat. 

The  profile  listing  was  produced  at  1991-08-14  13:32:48,  and  the  actual 
TCPRE  instrumentation  was  performed  at  1991-08-14  13:15:27. 

There  were  103  places  out  of  128  where  the  coverage  percentage  was 
below  the  selected  warning  level  100  %. 


Figure  E^6.  TCPOST  Coverage  Summary 
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APPENDIX  F 


SAMPLE  OUTPUTS  FROM  TestGen 


TABLE  F-1.  TestGen  Cyclomatic  Complexity  Report 


page  1  of  3 


Module  Name 

Design 

Code 

1  Ll_Support 

1  1 

1  1 

1 

1  Alternate 

1  1 

1  11 

1 

1  Merge_Ranges 

1  1 

1  2 

1 

1  Char_Range 

1  1 

1  3 

i 

1  Complete_Pat 

1  1 

1  12 

1 

1  Complete_Alt 

1  1 

1  8 

1 

1  Restrict 

1  1 

1  14 

1 

1  Tail 

1  1 

1  11 

1 

1  Resolve_Ambiguity 

1  1 

1  13 

1 

1  Complete_Concat 

1  1 

1  8 

1 

1  Complete_Opt 

1  1 

1  3 

1 

1  Complete_Patterns 

1  1 

1  2 

1 

1  Concatenate 

1  1 

1  3 

1 

1  Cvt_Ascii 

1  1 

1  10 

1 

1  Cvt_String 

1  1 

1  4 

Esa£sss  =  s=: 

1 

ssas 

TABLE  F-2.  TestGen  Test  Case  Effort  Report 


page  1  of  3 

Number  of  Test  Cases  Required  for 
Module  N2uae 

Basis 

Testing 

Branch  Full  Path 

Testing  Testing 

1  Ll_Support 

1  1 

1  1 

1  1 

1 

1  Alternate 

1  11 

1  7 

1  27 

1 

1  Merge_Ranges 

1  1 

1  1 

1  1 

1 

1  Char_Range 

1  2 

1  2 

1  2 

1 

1  Complete_Pat 

1  11 

1  11 

1  11 

1 

1  Complete_Alt 

1  7 

1  8 

1  8 

1 

1  Restrict 

1  13 

1  13 

1  13 

1 

1  Tail 

1  8 

1  8 

1  8 

I 

1  Resolve_Ambiguity 

1  12 

1  8 

1  36 

1 

1  Complete_Concat 

1  8 

1  8 

1  8 

1 

1  Complete_Opt 

1  3 

1  3 

1  3 

1 

1  Complete_Patterns 

1  1 

1  1 

j  1 

1 

1  Concatenate 

1  3 

1  3 

1  3 

1 
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*  Testing  all  paths  of  Subprogram:  Alternate 


Test  conditions  case  10  of  27  for  subprogreun:  Alternate 


Test  conditions  required  for  test  case  10  are: 


Set  (Right  =  null  or  else  Right .Variant  =  Bad  )  to  False 

Set  (Left .Variant  =  Bad  )  to  False 

Set  (Left. Variant  =  Alt  )  to  True 

Set  (Right. Variant  =  Alt  )  to  False 

Set  (New_Left .Variant  =  Rng  )  to  True 

Set  (New_Right .Variant  =  Rng  and  New_Lef t . Name  =  New_Right . Name  )  to  True 


Statements  to  be  executed  during  test  case  10  are: 


Procedure  Alternate  is 
Begin 

If  Right  =  null  or  else  Right .Variant  =  Bad  then 
***  Condition  is  False 
Elsif  Left. Variant  =  Bad 
***  Condition  is  False 

End  if  —  for  210 

If  Left. Variant  *  Alt  then 
***  Condition  is  True 
If  Right .Variant  =  Alt  then 
***  Condition  is  False 
Else 

New_Left  :=  Right; 

New_Right  :=  Left; 

End  if  —  for  216 

End  if  —  for  215 

If  New_Left . Variant  =  Rng  then 
•**  Condition  is  True 

If  New_Right . Variant  =  Rng  and  New_Lef t . Name  »  New_Right . Name 
***  Condition  is  True 
Return  Merge_Ranges ( New_Lef t , New_Right ) ; 

End 


Figure  F-1.  TestGen  Conditions  for  Path  Testing  ALTERNATE 
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